Criterion validity of the California Verbal Learning Test-Second Edition (CVLT-II) after traumatic brain injury

Archives of Clinical Neuropsychology 22 (2007) 143 149 Criterion validity of the California Verbal Learning Test-Second Edition (CVLT-II) after traumatic brain injury Monica L. Jacobs, Jacobus Donders Psychology Service, Mary Free Bed Rehabilitation Hospital, 235 Wealthy, S.E., Grand Rapids, MI 49503, United States Accepted 6 December 2006 Abstract The California Verbal Learning Test-Second Edition (CVLT-II) was performed by 200 participants, divided into moderate severe traumatic brain injury (n = 43), mild traumatic brain injury (n = 57) and demographically matched control (n = 100) groups. Participants with complicating premorbid histories or who scored below 15/16 on the CVLT-II forced choice recognition trial were excluded. There were statistically significant (p <.0001) effects of group status on the CVLT-II total recall discriminability and recognition discriminability indices. Logistic regression revealed that, in the classification of control versus moderate severe traumatic brain injury, CVLT-II variables were accurate 66 71% overall, but false positive rates ranged from 49 to 54%. In conclusion, average scores on the CVLT-II differ meaningfully between patients with various degrees of severity of traumatic brain injury and controls, but this test should not be used in isolation to determine the presence or absence of acquired memory impairment. 2006 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. Keywords: Learning; Memory; Traumatic brain injury The California Verbal Learning Test-Second Edition (CVLT-II; Delis, Kramer, Kaplan, & Ober, 2000) is an updated version of the original California Verbal Learning Test (CVLT; Delis, Kramer, Kaplan, & Ober, 1987), and is used to evaluate learning and memory in persons ranging in age from 16 to 89 years. The CVLT demonstrated sensitivity to a broad range of clinical conditions, and had seen especially widespread application in the evaluation of sequelae of traumatic brain injury (TBI; Crosson, Novack, Trenerry, & Craig, 1988; Curtiss, Vanderploeg, Spencer, & Salazar, 2001; Wiegner & Donders, 1999). Compared to its predecessor, innovations on the CVLT-II include the addition of a forced choice trial to assess level of effort, and the inclusion of recall discriminability indices, which are based not only on the number of correct words recalled but also take into account intrusions (i.e., words that were not from the original list). The reliability of the CVLT-II appears to be acceptable, with values of internal consistency ranging from.78 to.94, and the normative base is much more representative of the general U.S. population than that for the CVLT (Delis et al., 2000). The goal of the current study was to assess the criterion validity of the CVLT-II in patients with TBI. Criterion validity pertains to whether or not a psychometric variable of interest is meaningfully related to a relevant, external dimension or standard (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999). In this case, this has to do with whether or not Corresponding author. Tel.: +1 616 242 9201; fax: +1 616 451 9513. E-mail address: jacobus.donders@maryfreebed.com (J. Donders). 0887-6177/$ see front matter 2006 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.acn.2006.12.002

144 M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 scores on the CVLT-II are related to the presence and severity of brain injury. Several recent studies have provided support for the potential utility of the CVLT-II in various clinical samples. Baldo, Delis, Kramer, and Shimamura (2002) examined the CVLT-II profiles of 11 patients with focal frontal lobe lesions as compared to a control group. They found that the patients with frontal injuries learned fewer words, made more intrusion errors, and also had impaired yes/no recognition performance. Delis et al. (2005) compared the new recall discriminability indices to traditional scores (i.e., those based exclusively on target items correctly recalled) in the discrimination of patients with Huntington s disease and Alzheimer s disease. In their study, the recall discriminability indices distinguished the groups better, primarily because the patients with Alzheimer s disease made more intrusion errors, which affected the recall discriminability scores but not the traditional indices of accurate recall. Neither of these two studies addressed the sensitivity, specificity, or general predictive value of the CVLT-II. Although such data was not germane to the primary goal of those studies, information about this kind of classification is typically more important for clinical practice than knowledge about differences between group means (Ivnik et al., 2001). Furthermore, to our knowledge, no studies have examined the criterion validity of the CVLT-II in a large sample of patients with TBI. The purpose of the current investigation was to determine the extent of group differences in average performance on the CVLT-II between patients with various degrees of severity of TBI and demographically matched controls, and also to provide more detailed information about diagnostic classification accuracy when trying to predict group membership on the basis of the CVLT-II. It has been well established that deficits in learning and memory are fairly common after moderate to severe TBI, such as those injuries that are associated with prolonged loss of consciousness and/or acute intracranial lesions on neuroimaging (Hanks, Ricker, & Millis, 2004; Vakil, 2005). Both dominant temporal and prefrontal involvement may be especially detrimental in this regard (Baldo et al., 2002; Crosson, Sartor, Jenny, Nabors, & Moberg, 1993). However, there is some controversy about the possibility of persisting cognitive difficulties in persons with mild, uncomplicated TBI (Bigler, 2001; Mittenberg & Strauman, 2000; Ruff, Camenzuli, & Mueller, 1996). Most recent literature reviews suggest that although some deficits in learning and memory may occur early in the recovery from mild TBI (i.e., without coma and with negative neuroimaging findings), it is unusual to have such difficulties persist for extended periods of time in the absence of premorbid or comorbid complicating factors such as psychiatric history or financial compensation-seeking (Iverson, 2005; Larrabee, 2005; Schretlen & Shapiro, 2003). Based on the above-described literature, we hypothesized that (1) patients with moderate severe brain injury would have statistically significantly lower average scores on the CVLT-II than either patients with mild TBI or demographically matched controls, and (2) the average CVLT-II performance of patients with mild TBI would not be statistically significantly different from that of demographically matched controls. This was considered to be a first step in the establishment of criterion validity of the instrument. It was also determined a priori that, in order to be clinically useful as a tool in the classification of individual patients, the CVLT-II should have a likelihood ratio (sensitivity/[1 specificity]) 2. This has been suggested as a reasonable standard for clinical decision making, particularly when there are no strong reasons to value sensitivity disproportionately higher than specificity, or vice versa, on a routine basis with a particular medical diagnosis (Grimes & Schulz, 2005). 1. Method 1.1. Participants After receiving institutional review board approval, 100 clinical participants were obtained retrospectively from approximately four years of consecutive clinical referrals to a Midwestern rehabilitation facility. Candidates included in the study met the following criteria: (1) diagnosis of TBI through external force to the head with associated alteration of consciousness, (2) age between 17 and 80 years at the time of assessment, (3) evaluation with the CVLT-II within 1 year after injury, (4) absence of a premorbid history of special education, substance abuse, neurological impairment or psychiatric illness, (5) performance in the valid range (i.e., score of at least 15/16) on the forced choice recognition trial, a measure of effort and motivation (Moore & Donders, 2004), and (6) not currently involved in disputed financial compensation seeking. Only four potential participants who had not already been excluded for ongoing litigation or prior psychiatric history, were eliminated for failing the forced choice recognition trial of the CVLT-II. Three of them had sustained mild injuries, as defined below, and only one had extended length of coma. All of the remaining participants also completed and passed one or more of the following measures of effort: Word Memory Test, Test of Memory Malingering, and/or Reliable Digit Span. Only initial evaluations were used in this study. During the course

M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 145 Table 1 Demographic characteristics of control (n = 100), mild TBI (n = 57) and moderate severe TBI (n = 43) groups Variable Control Mild Moderate severe Age (years; M, SD) 35.25 (16.64) 37.33 (13.66) 32.09 (18.09) Days since injury (M, SD) 122.53 (87.12) 99.47 (61.38) Gender (%) Female 39.00 43.86 32.56 Male 61.00 56.14 67.44 Ethnicity (%) African 8.00 7.02 2.33 Asian 0.00 1.75 2.33 Caucasian 87.00 87.72 93.02 Latino 5.00 3.51 2.33 Education (%) 11 years 19.00 14.03 39.53 12 years 37.00 36.84 30.23 13 15 years 29.00 33.33 13.95 16 years 15.00 15.79 16.28 Note. TBI: traumatic brain injury. Control data are from the standardization sample of the California Verbal Learning Test-Second Edition (CVLT-II). Copyright 2000 by Harcourt Assessment, Inc. Used with permission. All rights reserved. of this investigation, the CVLT-II was routinely included in neuropsychological assessments of patients with TBI at the facility where this research was completed, except under circumstances that would have invalidated the test results (e.g., not fluent in English). Most of the clinical participants had been injured in a motor vehicle accident as a driver (n = 52), passenger (n = 23) or pedestrian (n = 2). Other injury conditions included falls (n = 9), recreational activities (n = 7), and other (n = 7). The average time since injury at the time of assessment was 112.61 days (SD = 77.61, range = 28 346). Several potential measures of injury severity were initially considered, including the Glasgow Coma Scale (GCS), length of posttraumatic amnesia (PTA), and length of coma. GCS scores were often affected by sedation, and tended to be quite variable in the first 24 h; making this an unstable index of injury severity. Length of PTA needed to be estimated retrospectively in a considerable proportion of the sample, and such estimates may not be reliable. For these reasons, injury severity was based on length of coma (defined as the number of days until verbal commands were followed), combined with results from neuroimaging of the brain in acute care, consistent with previously established criteria (Donders, Tulsky, & Zhu, 2001). Patients with moderate severe TBI (n = 43) had evidence of an acute intracranial lesion in neuroimaging studies (n = 38) or duration of coma of at least 24 h (n = 25), or both. Injuries were classified as mild (n = 57) if there was no evidence of acute intracranial lesions on neuroimaging studies and coma was less then 24 h. Twenty of the 47 participants in the moderate severe TBI group had also been included in a previous study that examined the psychometric characteristics of recall discriminability indices for the short and long, free and cued recall CVLT-II variables (Donders & Nienhuis, in press). That particular study was based on a much smaller sample, and did not include patients with mild TBI. After selection of the clinical sample, a demographically matched control group (n = 100) was obtained from the CVLT-II standardization sample. The control sample did not include any persons with self-reported neurological or psychiatric disorders (Delis et al., 2000). Background characteristics of the complete clinical and control samples are presented in Table 1. 1.2. Procedure The CVLT-II was administered according to standardized procedures to clinical participants as part of a comprehensive neuropsychological evaluation. Almost all of the participants had been assessed as outpatients, with evaluations done on an inpatient basis only when it would have been unfeasible for them to return for evaluation as an outpatient (e.g., living in a remote area without reliable access to transportation). All of the participants were evaluated only when they were medically stable and could recall meaningful information from day to day. Clinical patients 18 years

146 M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 and older provided informed consent, and participants under the age of 18 assented with the consent of their parents. CVLT-II raw scores were converted to age and gender corrected z (M =0,SD = 1) and T (M = 50, SD = 10) scores, using commercially available software (Delis & Fridlund, 2000). 1.3. Materials The CVLT-II is an individually administered list-learning task, which is used to examine several variables with regard to the ability to learn and remember verbally presented information. It also includes a forced choice trial administration that is intended to assess level of effort and motivation on the test. The main variables of interest for the purpose of this investigation were the following. The total A 1 5 T score reflects accurate recall over the five learning trials of the first list, and is most often used as a summary index of learning on the CVLT-II, with higher scores reflecting better performance. However, this index does not take into account recall errors. For this reason the z score for the total number of Intrusions across learning and recall trials was also considered. Higher scores reflect worse performance (i.e., more recall errors) on this variable. It was determined a priori that if there would be statistically significant group differences in terms of both accurate recall (total A 1 5 T score) and inaccurate recall (Intrusions), subsequent analysis of diagnostic classification accuracy would focus on the z scores for the recall discriminability indices, which consider the balance of accurate and inaccurate recall. The recall discriminability indices take into account the total number of words recalled on a trial, possible number of target words, the number of intrusions reported, and the number of possible intrusions (16). In this way, someone who has a large number of correct responses, but also has large number of intrusions will have a lower score then someone who has a large number of correct answers, but relatively few intrusions. Conversely, someone who recalls relatively few target words, but who also has relatively few intrusions will also have an impaired discriminability score, but it will not be as poor as that of someone who recalls few correct words and provides many intrusive errors (Delis et al., 2000). In addition, we planned to include in the analyses the z score for recognition discriminability index derived from the yes/no Recognition trial because it was considered possible that some patients might show improved performance under a recognition format, as opposed to free or semantically cued recall. Higher z scores reflect better performance for all discriminability variables. 1.4. Statistical analyses Mean group differences on the CVLT-II were evaluated with analysis of variance. In order to balance the risk of Type I and Type II errors with multiple comparisons, the Stepdown Bonferroni correction was applied to post hoc contrasts. Covariances between selected variables were calculated with the Pearson product-moment correlation coefficient. Finally, classification accuracy was evaluated by means of logistic regression analysis. 2. Results There were no statistically significant differences between the two clinical groups and the control group in terms of age, gender, ethnicity (dichotomized as Caucasian vs. other), or educational attainment (p >.05 for all variables). This suggests that they were adequately matched on demographic characteristics. The difference in time since injury between the two TBI groups was also not statistically significant, F (1, 98) = 2.19, p >.10, η 2 = 0.02. Since we did not include a minimum time since injury requirement in this study, we also computed correlations between time post injury and the four dependent variables of interest, total A 1 5, intrusions, total recall discriminability, and recognition discriminability, in the complete clinical sample. None of these correlations were statistically significant (p >.10 for all variables). Table 2 presents the CVLT-II results for the mild TBI, moderate severe TBI, and control groups. Findings from the immediate and delayed, free and cued recall discriminability indices yielded virtually the same results as those for the total recall discriminability index. For reasons of brevity and clarity, only results from total recall discriminability are reported here. Inspection of Table 2 suggests that the moderate severe TBI group had relatively worse performance across all CVLT-II variables than the control group, with the mild TBI group taking an intermediate position. There were statistically significant main effects of group for total A 1 5, F (2, 197) = 7.91, p <.0005, η 2 = 0.07, as well as for Intrusions, F (2, 197) = 8.83, p <.0002, η 2 = 0.08. Given this combination of findings, we followed through

M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 147 Table 2 Means and standard deviations of control (n = 100), mild TBI (n = 57) and moderate severe TBI (n = 43) groups for four CVLT-II variables Variable Control Mild Moderate severe Total A 1 5 T score (M, SD) 51.50 (9.06) 49.97 (11.65) 44.07 (11.5) Intrusions z score (M, SD) 0.04 (0.84) 0.40 (1.17) 0.85 (1.36) Total recall discriminability z score (M, SD) 0.09 (0.93) 0.26 (1.18) 0.90 (1.26) Recognition discriminability z score (M, SD) 0.18 (0.86) 0.31 (1.12) 0.61 (1.10) Note. TBI: traumatic brain injury. CVLT-II: California Verbal Learning Test-Second Edition. Control data are from the standardization sample of the California Verbal Learning Test-Second Edition (CVLT-II). Copyright 2000 by Harcourt Assessment, Inc. Used with permission. All rights reserved. Table 3 Classification accuracy (%) of selected CVLT-II variables CVLT-II variable Sensitivity Specificity False positives False negatives Total recall discriminability 74.42 63.00 53.62 14.87 Recognition discriminability 60.47 75.00 49.02 18.48 Note. CVLT-II: California Verbal Learning Test-Second Edition. on our original plan to focus subsequent analyses on the total recall discriminability index, F (2, 197) = 12.48, p <.0001, η 2 = 0.11, and the recognition discriminability index F (2, 197) = 10.56, p <.0001, η 2 = 0.10. Post-hoc contrasts with the Stepdown Bonferroni correction suggested that: (a) the moderate severe TBI group performed statistically significantly worse than the control group on both total recall discriminability, F (1, 141) = 26.91, p <.0006, η 2 = 0.16, and recognition discriminability, F (1, 141) = 20.87, p <.0008, η 2 = 0.13; (b) the performance of the moderate severe TBI group was also statistically significantly worse than that of the mild TBI group on total recall discriminability, F (1, 98) = 6.61, p <.04, η 2 = 0.06, but not on recognition discriminability, F (1, 98) = 1.76, p >.10, η 2 = 0.02; and (c) the mild TBI group performed statistically significantly worse than the control group on recognition discriminability, F (1, 155) = 9.14, p <.02, η 2 = 0.06, but not on total recall discriminability, F (1, 155) = 4.26, p <.09, η 2 = 0.03. Because these two CVLT-II variables yielded somewhat different results, we evaluated their covariances in the clinical and control samples, as well as their respective sensitivities to length of coma in the complete clinical sample (n = 100). Total recall discriminability and recognition discriminability were strongly correlated in patients with TBI (r =.72, p <.0001), and also in the control participants (r =.66, p <.0001); sharing, respectively, 51.84% and 43.56% of the variance. However, in the complete clinical sample, total recall discriminability demonstrated a statistically significant correlation with length of coma, r =.21, p <.04, whereas recognition discriminability did not, r =.14, p >.10. For these reasons, we decided to include both variables in the subsequent analyses. We then used logistic regression analysis to determine how accurately these CVLT-II variables could classify individuals into the moderate severe TBI (n = 43) and control (n = 100) groups. Analyses were run separately for total recall discriminability and recognition discriminability. These results are presented in Table 3. Total recall discriminability predicted group membership correctly 66.43% of the time, with a likelihood ratio of 2.01, whereas recognition discriminability classified 70.63% of these participants correctly, with a likelihood ratio of 2.42. These differences in total accuracy percentages were not statistically significant (z = 0.50, p >.10). Both variables were associated with relatively low false negative rates but fairly high false positive rates. 3. Discussion The purpose of this investigation was to determine the clinical utility of the CVLT-II in differentiating between patients with varying degrees of severity of TBI, and in discriminating such patients from demographically matched healthy controls. The results suggest that CVLT-II variables were moderately effective for this purpose at a group level but not sufficiently accurate on an individual classification basis. Our first hypothesis was largely confirmed. Patients with moderate severe TBI indeed obtained statistically significantly lower mean scores than the control group on both total recall discriminability and recognition discriminability. Furthermore, compared to the mild TBI group, the moderate severe TBI group also performed worse, although this

148 M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 difference was statistically significant only for total recall discriminability. Thus, in terms of mean group differences, especially the total recall discriminability index appeared to have considerable criterion validity. Support for our second hypothesis was more equivocal. As predicted, the average performance of patients with mild TBI was not statistically different from that of the control group on total recall discriminability. However, contrary to expectation, there was a statistically significant difference between the mild TBI group and the control group on recognition discriminability, in favor of the latter group. It should be noted that this was a relatively small difference, and also that the performance of the mild TBI group on this variable was still within less than half a standard deviation from the normative mean. Furthermore, recognition discriminability did not correlate to a statistically significant degree with length of coma in the complete clinical sample, whereas total recall discriminability did. For these reasons, we conclude that the mean difference between the mild TBI group and the control group on recognition discriminability is of uncertain clinical significance. Replication in a much larger sample would be needed to determine if this finding might reflect the presence of a small subgroup of people with mild TBI with subtle cognitive residuals. With regard to the classification accuracy of the CVLT-II variables, we had set an a priori criterion that, in order to be clinically significant, the CVLT-II should have a likelihood ratio 2. When discriminating between patients with moderate severe TBI and control participants, both total recall discriminability and recognition discriminability met this criterion, suggesting that these CVLT-II variables have the minimal required value in the diagnostic process. However, a concerning finding was that both variables were associated with large proportions of false positives. In fact, approximately half of all the positive classifications (37/69 for total recall discriminability, and 25/51 for recognition discriminability) applied to individuals from the demographically matched control group. This is likely related to the fact that, as seen in Table 2, there was a considerable amount of overlap in the scores between the control group and the moderate severe TBI group. Thus, in clinical circumstances where the base rate of moderate severe TBI is similar to that in the current study (0.30; 43/143), the CVLT-II may have adequate negative predictive value, but problematic positive predictive value. Of course, it needs to be realized that negative and positive predictive accuracies are strongly affected by base rates (Rosenfeld, Sands, & Van Gorp, 2000). Lower base rates would be associated with even higher false positive rates. This is concerning because it is estimated that, of all cases of TBI that occur annually, only about 10 20% involve moderate or severe injuries (Centers for Disease Control and Prevention, 2006; Thurman, Alverson, Dunn, Guerrero, & Sniezek, 1999). Our findings suggest that total recall discriminability is the CVLT-II variable with the clearest criterion validity in terms of sensitivity to severity of TBI. However, this variable misclassified an unacceptably large number of the demographically matched control participants. For these reasons, the CVLT-II should not be used in isolation to determine the likelihood of acquired neurocognitive dysfunction. This caution may be especially important in medicolegal contexts involving cases of mild TBI, where other complicating factors are often influential (Binder & Rohling, 1996; Green, Rohling, Lees-Haley, & Allen, 2001). Potential limitations of the present investigation should also be considered. The clinical participants were recruited from a referred convenience sample at a rehabilitation hospital. This may have led to the inclusion of relatively more patients with serious neurological injuries than if they had been selected from consecutive emergency room admissions. At the same time, this also guaranteed a broad range of injury severity, which reduced the probability of unreliable findings due to restriction of range. The clinical sample was also limited largely to Caucasian individuals from the Midwest; therefore, replication with a more ethnically and geographically diverse sample would be desirable. With these limitations in mind, the findings from this investigation indicate that when combined with additional neuropsychological test data, patient history and behavioral observations, the CVLT-II may be clinically useful. However, due to low positive predictive accuracy, the CVLT-II should never be used in isolation to determine the presence or absence of acquired memory impairment or brain injury. Future research should investigate the possibility of different subtypes of impairment of learning and memory, as assessed by the CVLT-II, after TBI. Another goal for future research is to determine the degree to which CVLT-II variables add incremental value (above neurological and demographic variables) in the prediction of longer-term outcome after TBI. Acknowledgements This research was supported by a grant from the Campbell Foundation and was based in part on standardization data of the California Verbal Learning Test-Second Edition (CVLT-II). Copyright 2000 by Harcourt Assessment, Inc. Used with permission. All rights reserved. California Verbal Learning Test and CVLT are trademarks of Harcourt

M.L. Jacobs, J. Donders / Archives of Clinical Neuropsychology 22 (2007) 143 149 149 Assessment, Inc.; registered in the United States of America and other jurisdictions. The authors thank the publisher for access to these standardization data. References American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Baldo, J. V., Delis, D., Kramer, J., & Shimamura, A. P. (2002). Memory performance on the California Verbal Learning Test-II: Findings from patients with focal frontal lesions. Journal of the International Neuropsychological Society, 8, 539 546. Bigler, E. D. (2001). The lesion(s) in traumatic brain injury: Implications for clinical neuropsychology. Archives of Clinical Neuropsychology, 16, 95 131. Binder, L. M., & Rohling, M. L. (1996). Money matters: A meta-analytic review of the effects of financial compensation on recovery after closed head injury. American Journal of Psychiatry, 153, 7 10. Centers for Disease Control and Prevention. (2006). Incidence rates of hospitalization related to traumatic brain injury 12 states, 2002. Morbidity and Mortality Weekly Report, 55, 201 204. Crosson, B., Novack, T. A., Trenerry, M. R., & Craig, P. L. (1988). California Verbal Learning Test (CVLT) performance in severely head-injured and neurologically normal adult males. Journal of Clinical and Experimental Neuropsychology, 10, 754 768. Crosson, B., Sartor, K. J., Jenny, A. B., Nabors, N. A., & Moberg, P. J. (1993). Increased intrusions during verbal recall in traumatic and nontraumatic lesions of the temporal lobe. Neuropsychology, 7, 193 208. Curtiss, G., Vanderploeg, R. D., Spencer, J., & Salazar, A. M. (2001). Patterns of verbal learning and memory in traumatic brain injury. Journal of International Neuropsychological Society, 7, 574 585. Delis, D. C., & Fridlund, A. J. (2000). CVLT-II comprehensive scoring program. San Antonio, TX: Psychological Corporation. Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (1987). California Verbal Learning Test. San Antonio, TX: Psychological Corporation. Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (2000). California Verbal Learning Test (2nd ed.). San Antonio, TX: Psychological Corporation. Delis, D. C., Wetter, S. R., Jacobson, M. W., Peavy, G., Hamilton, J., Gongvatana, A., et al. (2005). Recall discriminability: Utility of a new CVLT-II measure in the differential diagnosis of dementia. Journal of the International Neuropsychological Society, 11, 708 715. Donders, J., & Nienhuis, J. B. (in press). Utility of CVLT II recall discriminability indices in the evaluation of traumatic brain injury. Journal of the International Neuropsychological Society. Donders, J., Tulsky, D. S., & Zhu, J. (2001). Criterion validity of new WAIS-III subtest scores after traumatic brain injury. Journal of the International Neuropsychological Society, 7, 892 898. Green, P., Rohling, M. L., Lees-Haley, P. R., & Allen, L. M. (2001). Effort has a greater effect on test scores than severe brain injury in compensation claimants. Brain Injury, 15, 1045 1060. Grimes, D. A., & Schulz, K. F. (2005). Refining clinical diagnoses with likelihood ratios. Lancet, 365, 1500 1505. Hanks, R. A., Ricker, J. H., & Millis, S. R. (2004). Empirical evidence regarding the neuropsychological assessment of moderate and severe traumatic brain injury. In J. H. Ricker (Ed.), Differential diagnosis in adult neuropsychological assessment (pp. 218 242). New York: Springer. Iverson, G. L. (2005). Outcome from mild traumatic brain injury. Current Opinions in Psychiatry, 18, 301 317. Ivnik, R. J., Smith, G. E., Cerhan, J. H., Boeve, B. F., Tangalos, E. G., & Peterson, R. C. (2001). Understanding the diagnostic capabilities of cognitive tests. The Clinical Neuropsychologist, 15, 114 124. Larrabee, G. J. (2005). Mild traumatic brain injury. In G. J. Larrabee (Ed.), Forensic neuropsychology: A scientific approach (pp. 209 236). New York: Oxford. Mittenberg, W., & Strauman, S. (2000). Diagnosis of mild head injury and the postconcussion syndrome. Journal of Head Trauma Rehabilitation, 15, 783 791. Moore, B. A., & Donders, J. (2004). Predictors of invalid neuropsychological test performance after traumatic brain injury. Brain Injury, 18, 975 984. Rosenfeld, B., Sands, S. A., & Van Gorp, W. G. (2000). Have we forgotten the base rate problem? Methodological issues in the detection of distortion. Archives of Clinical Neuropsychology, 15, 349 359. Ruff, R. M., Camenzuli, L. F., & Mueller, J. (1996). Miserable minority: Emotional risk factors that influence the outcome of a mild traumatic brain injury. Brain Injury, 10, 551 565. Schretlen, D. J., & Shapiro, A. M. (2003). A quantitative review of the effects of traumatic brain injury on cognitive functioning. International Review of Psychiatry, 15, 341 349. Thurman, D. J., Alverson, C., Dunn, K. A., Guerrero, J., & Sniezek, J. E. (1999). Traumatic brain injury in the United States: A public health perspective. Journal of Head Trauma Rehabilitation, 14, 602 615. Vakil, E. (2005). The effect of moderate to severe traumatic brain injury (TBI) on different aspects of memory: A selective review. Journal of Clinical and Experimental Neuropsychology, 27, 977 1021. Wiegner, S., & Donders, J. (1999). Performance on the California Verbal Learning Test after traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 21, 159 170.