Performance profiles and cut-off scores on the Memory Assessment Scales

Archives of Clinical Neuropsychology 19 (2004) 489 496 Performance profiles and cut-off scores on the Memory Assessment Scales Sid E. O Bryant a, Kevin Duff b, Jerid Fisher c, Robert J. McCaffrey a,d, a Department of Psychology, University at Albany, State University of New York, Albany, NY 12222, USA b University of Oklahoma Health Sciences Center, USA c Brain Injury Consultants, Rochester, NY, USA d Albany Psychological Associates, P.C., 1740 Western Avenue, Albany, NY 12203, USA Abstract Accepted 17 July 2003 The increased role of neuropsychologists in the courtroom has led to an increased effort in the detection of possible symptom exaggeration/malingering. Whereas domain specific measures of malingering have traditionally been used in this detection process, the identification of performance profiles and cut-off scores on standard neuropsychological assessment instruments may provide an alternate strategy. The present study evaluated the effectiveness of performance profiles and cut-off scores in discriminating traumatic brain injury (TBI) litigants suspected of malingering from those not suspected of malingering on the Memory Assessment Scales (MAS). Results suggest that TBI litigants suspected of poor effort will perform globally at a lower level than TBI litigants not suspected of poor effort on nearly all MAS indices, however, the performance profiles of each group was similar. Cut-off scores, especially when used in combination, were also effective in correctly classifying individuals in the two groups. The present findings warrant further research examining the utility of the proposed cut-off scores separately and concomitantly. Such research will aid the clinical neuropsychological practitioner in interpreting aberrant performance profiles on the MAS in forensic situations. 2003 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. Keywords: Memory Assessment Scales; Malingering; TBI litigants; Test of Memory Malingering Portions of this data were presented at the 20th Annual meeting of the National Academy of Neuropsychology in Orlando, FL. Corresponding author. Tel.: +1-518-442-4841; fax: +1-518-442-5098. E-mail address: rm188@albany.edu (R.J. McCaffrey). 0887-6177/$ see front matter 2003 National Academy of Neuropsychology. doi:10.1016/j.acn.2003.07.001

490 S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 The increased role of clinical neuropsychological practitioners in forensic situations has led to an increased effort in the search for means of accurately identifying symptom exaggeration/malingering. This research has typically taken one of two paths: the creation of domain specific tests designed specifically for the detection of possible malingering, or the identification of performance patterns and cut-off scores on standard neuropsychological instruments. Domain specific tests of malingering, which include the Rey 15-Item test (Rey-15, Rey, 1964), the Test of Memory Malingering (TOMM, Tombaugh, 1996), and the Victoria Symptom Validity Test (VSVT, Slick, Hopp, & Strauss, 1995), have been shown to be valuable additions to the forensic neuropsychological evaluation (Bernard & Fowler, 1990; Rees, Tombaugh, & Boulay, 2001; Rees, Tombaugh, Gansler, & Moczynski, 1998; Tombaugh, 1996, 1997). Alternatively, researchers have attempted to detect symptom exaggeration using established neuropsychological instruments (Donders, 1999; Iverson, Franzen, & McCracken, 1994; Millis, 1992, 1994; Mittenberg, Theroux-Fichera, Zielinski, & Heilbronner, 1995; Tenhula & Sweet, 1996). Standard scoring systems for these instruments, however, have demonstrated a limited ability in discriminating neurologically impaired individuals from those suspected of malingering (Faust, Hart, & Guillmette, 1988; Greiffenstein, Baker, & Gola, 1994; Heaton, Lehman, & Vogt, 1978). Supplementary scoring systems, such as performance profiles and/or cut-off scores may allow these instruments to more accurately make these distinctions. Utilizing performance profiles and/or cut-off scores would have the additional advantage of saving time. Unfortunately, there is little empirical evidence to support the use of performance profiles and cut-off scores for many neuropsychological measures. Erroneous beliefs about brain injury and the resulting memory sequelae are prevalent in laypersons (Aubrey, Dobbs, & Rule, 1989; Nies & Sweet, 1994) and as a result, when looking for instruments that may aid in the identification of insufficient effort, memory measures are a logical choice. Therefore, it is likely that possible malingerers will perform poorly on standardized tests of memory, such as the Memory Assessment Scales (MAS, Williams, 1991). Beetar and Williams (1995) examined the performance of simulators on the MAS and observed that the simulators generally performed more poorly than non-simulators on the MAS. This study has yet to be replicated in a sample of traumatic brain injury (TBI) litigants. The purpose of the present study was to evaluate the effectiveness of performance profiles and cut-off scores in discriminating TBI litigants suspected of symptom exaggeration from those not suspected of symptom exaggeration on the MAS. 1. Method 1.1. Participants Archival data was collected on 97 adult litigants (46 men and 51 women) referred to private neuropsychological practices for TBI evaluation. Each litigant fit Ruff & Richardson, 1999 classification of Mild TBI. Litigants were each referred by their attorneys, treating neurologists, or primary care physician. Based on the participant s performance on the TOMM and/or the Rey-15, they were grouped, as either those suspected of symptom exaggeration/malingering (malingering group, n = 29) or those not suspected of symptom exaggeration/malingering

S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 491 Table 1 Descriptive statistics for the malingering and non-malingering groups Malingering group (N = 29) Non-malingering group (N = 68) Age 44.3 (11.5) 39.6 (12.5) Education 12.1 (2.0) 13.5 (2.8) Gender a 16/13 30/38 TOMM 1 28.6 (7.9) 45.7 (4.0) TOMM 2 32.9 (9.3) 48.9 (1.6) TOMM 3 33.5 (8.7) 49.0 (1.3) Rey-15 8.1 (3.6) 13.2 (3.6) a Gender: number of males/number of females. (non-malingering group, n = 68). Demographic information for each sample is presented in Table 1. 1.2. Measures 1.2.1. Memory Assessment Scales (MAS, Williams, 1991) The MAS is a memory battery consisting of 12 subtests that assess three general areas of cognitive function: attention, concentration, and short-term memory; learning and immediate memory; and delayed memory (Beetar & Williams, 1995). The MAS assesses both verbal and non-verbal/visual memory and has been shown to be both a reliable and valid indicator of memory functioning (Lezak, 1995). 1.2.2. Test of Memory Malingering (TOMM, Tombaugh, 1996) The TOMM is a visual recognition task developed for identification of suspected malingering. An examinee is suspected of malingering if she/he scores less than 45 on Trial 2 or 3 (Tombaugh, 1996). The TOMM has been found to be relatively unaffected by age, education, psychiatric conditions, and a variety of neurological conditions known to cause genuine memory impairment, and has demonstrated high levels of sensitivity and specificity (Rees et al., 1998, 2001; Tombaugh, 1996, 1997). 1.2.3. Rey 15-Item Test (Rey-15, Rey, 1964) The Rey-15 is a visual recall task in which 15 items are presented for 10 s in a 3 5 array. The stimuli are removed, and the participant is immediately asked to reproduce the stimuli. Poor performance is said to be suggestive of malingering when less than nine items are correctly reproduced (Spreen & Strauss, 1998). 1.3. Procedures The MAS, TOMM, and Rey-15 were administered to all participants in the context of a larger neuropsychological evaluation. Standard administration protocols were followed. Participants were divided into two groups based on their performance on the TOMM and/or Rey-15 based

492 S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 on the cut-off scores outlined above. First groups were compared to one another on all MAS Summary Scores and subtests. Next, to determine cut-off scores, the 16 MAS scores (4 summary, 12 subtests) were converted to z-scores using normative information provided in the manual. Sensitivity, specificity, and hit rates were calculated for each score comparing the two groups. 2. Results Demographic information for both groups can be found in Table 1. The mean age for the entire sample was 41.1 years (S.D. = 12.3) and the mean number of years of education was 13.1 (S.D. = 2.7). Based on the TOMM and/or Rey-15 performance, there were 29 suspected malingerers (malingering group) and 68 non-malingerers (non-malingering group). There was no significant difference between the malingering and non-malingering group in age, t(95) = 1.75, P>.05. There was a significant difference in education, t(95) = 2.83, P<.01, but it was relatively small (12.1 vs. 13.5). There was no significant difference in gender, χ 2 (1) = 1.49, P>.05. Of those patients suspected of malingering, 19 were suspected on TOMM scores alone, 3 on Rey-15 scores alone, and 7 on TOMM and Rey-15 scores. In order to determine the comparability of these three groups, they were compared on demographic variables and MAS variables. There were no significant differences between these groups on age, F(2, 27) = 2.04, P =.15; education, F(2, 27) = 0.95, P =.40; or gender, F(2, 27) = 1.43, P =.26. There was no significant main effect for group (i.e., suspected of malingering based on TOMM only, Rey-15 only, or TOMM and Rey-15), F(18, 32) = 0.96, P =.56. Therefore all three groups were collapsed and analyzed as one suspected malingering group. Descriptive and inferential statistics for MAS scores by group are shown in Table 2. Two separate MANCOVAs were conducted, using education as a covariate: one analyzing the 4 Summary Scores and another analyzing the 12 subtests. There was a significant main effect for group (malingering vs. non-malingering) on the Summary Scores, F(4, 91) = 11.49, P<.001. Follow-up analyses indicated that the malingering group performed significantly below the non-malingering group on all four MAS Summary Scores (see Table 2). The second MANCOVA yielded a significant main effect for group (malingering vs. non-malingering) for MAS subtest scores, F(12, 83) = 4.53, P<.05. Follow-up analyses indicated that the malingering group scored significantly below (P <.05) the non-malingering group on all 12 subtests (see Table 2). In order to further evaluate the ability of the MAS to discriminate between malingering and non-malingering individuals, cut-off scores were created for each score. These cut-off scores, and their respective sensitivity, specificity, and hit rate values are presented in Table 3. Figure 1 was created in order to visually demonstrate the obtained performance patterns on the MAS global scores. It is clear that it is the level of performance, not the pattern of performance that is different between the two groups. To further illustrate this point the groups utilized in the Beetar and Williams (1995) study were included into this figure. The performance pattern of all groups were similar, however, the level of performance was not. The same results were also found for the subscale scores.

S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 493 Table 2 MAS scores of malingering and non-malingering groups MAS score Non-malingering group Malingering group F(1, 95) STM 89.3 (16.3) 67.6 (14.0) 38.7 VeM 89.6 (17.5) 68.8 (10.3) 34.4 ViM 95.7 (16.6) 77.6 (15.0) 26.8 GM 91.2 (18.0) 69.0 (11.0) 37.9 VeSp 7.1 (3.3) 3.4 (3.1) 26.9 ViSp 8.4 (2.8) 5.1 (2.4) 30.7 LA 7.3 (3.6) 3.8 (2.4) 20.3 LR 7.2 (3.9) 2.9 (2.7) 26.4 DLR 6.4 (3.7) 2.5 (2.5) 25.2 IPR 8.1 (3.4) 4.6 (2.4) 26.7 DPR 8.3 (3.7) 4.5 (2.3) 26.7 INF 7.3 (3.7) 3.7 (2.4) 22.5 DNF 7.1 (3.3) 4.7 (3.1) 11.4 VP 9.2 (3.3) 6.0 (2.8) 21.2 IVR 8.4 (2.8) 5.7 (2.7) 19.7 DVR 7.6 (4.2) 5.4 (4.3) 5.3 Note. MAS: Memory Assessment Scales, STM: Short-Term Memory, VeM: Verbal Memory, ViM: Visual Memory, GM: Global Memory, VeSp: Verbal Span, ViSp: Visual Span, LA: List Acquisition, LR: List Recall, DLR: Delayed List Recall, IPR: Immediate Prose Recall, DPR: Delayed Prose Recall, INF: Immediate Names and Faces, DNF: Delayed Names and Faces, VP: Visual Production, IVR: Immediate Visual Recognition, DVR: Delayed Visual Recognition. P =.023. P =.001. P<.001. Fig. 1. MAS mean global scores for malingering and non-malingering groups from present study and control and simulator groups from Beetar and Williams (1995). Note: Non-Malingering and Malingering groups are from data collected in the present study; Controls and Simulators are from Beetar and Williams (1995). 3. Discussion The present study was conducted in an attempt to identify performance patterns and cut-off scores that would be useful in discriminating between TBI litigants suspected of symptom

494 S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 Table 3 Cut-off scores for the Memory Assessment Scales and their associated hit rates, sensitivity, specificity, positive predictive power, and negative predictive power MAS score Cut-off a Hit rate (%) Sens Spec PPP NPP STM 2.00 81 0.64 0.87 0.69 0.87 VeM 1.50 78 0.86.75.59.93 ViM 1.20 82 0.75.85 0.68 0.88 GM 1.93 82 0.75 0.85 0.68.89 VeSp 2.00 76 0.65 0.81 0.59 0.85 ViSp 2.00 77 0.38 0.94 0.73 0.78 LA 2.33 76 0.55 0.85 0.61 0.82 LR 2.33 78 0.76 0.79 0.61 0.88 2.00 78 0.86 0.75 0.59 0.93 DLR 2.00 75 0.83 0.72 0.56 0.91 IPR 2.33 76 0.34 0.94 0.71 0.77 DPR 2.33 73 0.34 0.90 0.59 0.76 INF 3.00 72 0.31 0.90 0.56 0.75 2.33 72 0.55 0.79 0.53 0.80 DNF 3.00 72 0.14 0.97 0.67 0.72 2.67 72 0.27 0.91 0.57 0.75 VP 1.67 78 0.43 0.94 0.75 0.80 IVR 2.33 74 0.27 0.94 0.67 0.75 DVR 3.00 74 0.25 0.94 0.64 0.75 Note. MAS: Memory Assessment Scales, Sens: sensitivity, Spec: specificity, PPP: positive predictive power, NPP: negative predictive power, CVHR: cross validation sample hit rate, STM: Short-Term Memory, VeM: Verbal Memory, ViM: Visual Memory, GM: Global Memory, VeSp: Verbal Span, ViSp: Visual Span, LA: List Acquisition, LR: List Recall, DLR: Delayed List Recall, IPR: Immediate Prose Recall, DPR: Delayed Prose Recall, INF: Immediate Names and Faces, DNF: Delayed Names and Faces, VP: Visual Production, IVR: Immediate Visual Recognition, DVR: Delayed Visual Recognition. a Scores were converted to z-scores and cut-off scores represent standard deviation units below the mean. exaggeration from those not suspected of symptom exaggeration. Results demonstrated that there were in fact significant differences between the present malingering and non-malingering groups on all MAS scores, supporting the results of Beetar and Williams (1995). Next, visual comparisons were conducted utilizing both the present sample of TBI litigants and those utilized in the Beetar and Williams (1995) study. This comparison showed that while the pattern of performance remained similar between all groups the level of performance differed between groups. These results support previous research suggesting that the level of performance achieved by suspected malingerers on standard neuropsychological instruments is often lower than that of individuals not suspected of malingering (van Gorp et al., 1999; O Bryant, Hilsabeck, Fisher, & McCaffrey, 2003; Suhr, Tranel, Wefel, & Barrash, 1997). Next, the present results suggest that cut-off scores on the MAS may be useful in the identification of poor effort in mild TBI litigants and further research is warranted. Cut-off scores, which typically fell two standard deviations below the normative samples mean, were useful in correctly classifying 70 80% of the litigant sample. Although the sensitivity, specificity, and predictive powers of individual measures did not achieve optimal classification rates, various combinations of cut-off scores, used serially, were quite accurate. For example, if

S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 495 a cut-off score of 2 for the Visual Span subtest was applied to the sample, 77% of participants were correctly classified; and if a cut-off of 2 was applied to List Recall, 78% were correctly grouped. If, however, these two cut-off scores were used in combination serially (i.e., apply Visual Span s cut-off to all the participants, then apply List Recall s cut-off to all suspected malingerers based on Visual Span), the hit rate improved to 96%. Although cross-validation of these serial combinations is needed, they provide promising alternative strategies for detecting malingering. The present results suggest that the level of performance achieved by malingerers on the MAS is generally suppressed. While poor performance on the MAS in itself is not indicative of malingering, extremely poor performance across nearly all MAS indices may aid in the detection of possible symptom exaggeration/malingering when used in conjunction with significant findings on domain specific tests such as the TOMM, Rey-15, and VSVT, as well as information gathered from secondary sources and clinical interviews. Similarly, the use of cut-off scores, especially when used in combination serially, may assist the clinician in deciphering poor abilities from poor effort. It is hoped that the present results will serve as an impetus for further research examining the utility of cut-off scores on the MAS that may be suggestive of symptom exaggeration/malingering. Once validated, the present results will aid the clinical neuropsychological practitioner in the evaluation and interpretation of forensic cases. References Aubrey, J. B., Dobbs, A. R., & Rule, B. G. (1989). Layperson s knowledge about the sequelae of minor head injury and whiplash. Journal of Neurology, Neurosurgery, and Psychiatry, 52, 842 846. Beetar, J. T., & Williams, M. (1995). Malingering response styles on the Memory Assessment Scales and symptom validity tests. Archives of Clinical Neuropsychology, 10, 57 72. Bernard, L. C., & Fowler, W. (1990). Assessing the validity of memory complaints: Performance of brain damaged and normal individuals on Rey s task to detect malingering. Journal of Clinical Psychology, 46, 432 436. Donders, J. (1999). Specificity of malingering formula for the Wisconsin Card Sorting Test. Journal of Forensic Neuropsychology, 1, 35 42. Faust, D., Hart, K. J., & Guilmette, T. J. (1988). Pediatric malingering: The capacity of children to fake believable deficits on psychological testing. Journal of Consulting and Clinical Psychology, 36, 578 582. van Gorp, W. G., Humphrey, L. A., Kalechstein, A., Brumm, V. L., McMullen, W. J., Stoddard, M., & Pachana, N. A. (1999). How well do standard clinical neuropsychological tests identify malingering? A preliminary analysis. Journal of Clinical and Experimental Neuropsychology, 21, 245 250. Greiffenstein, M. F., Baker, J. W., & Gola, T. (1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6, 218 224. Heaton, R. K., Smith, H. H., Lehman, R. A., & Vogt, A. T. (1978). Prospects for faking believable deficits on neuropsychological testing. Journal of Clinical and Consulting Psychology, 46, 892 900. Iverson, G. L., Franzen, M. D., & McCracken, L. M. (1994). Application of a forced choice memory procedure designed to detect experimental malingering. Archives of Clinical Neuropsychology, 9, 437 450. Lezak, M. D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press. Millis, S. R. (1992). The Recognition Memory Test in the detection of malingered and exaggerated memory deficits. The Clinical Neuropsychologist, 6, 406 414. Millis, S. R. (1994). Assessment of motivation and memory with the Recognition Memory Test after financially compensable mild head injury. Journal of Clinical Psychology, 50, 601 605.

496 S.E. O Bryant et al. / Archives of Clinical Neuropsychology 19 (2004) 489 496 Mittenberg, W., Theroux-Fichera, S., Zielinski, R. E., & Heilbronner, R. L. (1995). Identification of malingered head injury on the Wechsler Adult Intelligence Scales Revised. Professional Psychology: Research and Practice, 26, 491 498. Nies, K. J., & Sweet, J. J. (1994). Neuropsychological assessment and malingering: a critical review of past and present strategies. Archives of Clinical Neuropsychology, 9, 501 552. O Bryant, S. E., Hilsabeck, R. C., Fisher, J. M., & McCaffrey, R. J. (2003). Utility of the Trail Making Test in the assessment of malingering in a sample of mild traumatic brain injury litigants. The Clinical Neuropsychologist, 17, 69 74. Rees, L. M., Tombaugh, T. N., & Boulay, L. (2001). Depression and the Test of Memory Malingering. Archives of Clinical Neuropsychology, 16, 501 506. Rees, L. M., Tombaugh, T. N., Gansler, D. A., & Moczynski, N. P. (1998). Five validation experiments of the Test of Memory Malingering (TOMM). Psychological Assessment, 10, 10 20. Rey, A. (1964). L examen clinique en psychologie. Paris: Presses Universitaires de France. Ruff, R.M. & Richardson, A.M., (1999). Mild traumatic brain injury. In: J.J. Sweet (Ed.). Forensic neuropsychology: Fundamentals and practice. Studies in neuropsychology, development, and cognition (pp. 315 338). Slick, D. J., Hopp, G. A., & Strauss, E. H. (1995). The Victoria Symptom Validity Test. Odessa, FL: Psychological Assessment Resources. Spreen, O., & Strauss, E. (1998). A Compendium of neuropsychological tests. New York: Oxford University Press. Suhr, J., Tranel, D., Wefel, J., & Barrash, J. (1997). Memory performance after head injury: Contributions of malingering, litigation status, psychological factors, and medication use. Journal of Clinical and Experimental Neuropsychology, 19, 500 514. Tenhula, W. N., & Sweet, J. J. (1996). Double cross-validation of the Booklet Category Test in detecting malingered traumatic brain injury. The Clinical Neuropsychologist, 10, 104 116. Tombaugh, T. N. (1996). Test of Memory Malingering. Toronto, Canada: Multi-Health Systems. Tombaugh, T. N. (1997). The Test of Memory Malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals. Psychological Assessment, 9, 260 268. Williams, J. M. (1991). Memory Assessment Scales. Odessa, FL: Psychological Assessment Resources.