Base Rates of Impaired Neuropsychological Test Performance Among Healthy Older Adults

Transcription

1 Archives of Clinical Neuropsychology, Vol. 13, No. 6, pp , 1998 Copyright 1998 National Academy of Neuropsychology Printed in the USA. All rights reserved /98 $ PII S (97) Base Rates of Impaired Neuropsychological Test Performance Among Healthy Older Adults Barton W. Palmer, Kyle B. Boone, Ira M. Lesser, and Marcy A. Wohl Harbor-UCLA Medical Center Base rates of deficient neuropsychological test performance were evaluated among 132 neurologically healthy older normal adults using a variety of measures commonly employed in the flexiblebattery approach to neuropsychological assessment. Subjects were divided into three age groups (50 to 59, 60 to 69, and 70 to 79 years). Despite the healthy status of our sample, most tests yielded at least some proportion of subjects earning scores in the borderline and impaired ranges (1.3 and 2.0 standard deviations below the age-group mean, respectively). Across the battery of measures, 73% of subjects earned a borderline score on at least one measure, and 20% of subjects earned at least two scores in the impaired range on separate tests. The proportion of subjects consistently earning borderline or impaired scores across multiple measures within specific cognitive domains was generally lower. Results illustrate the problems in interpreting isolated low scores, and the need to consider false-positive base rates in drawing inferences from poor test performance National Academy of Neuropsychology. Published by Elsevier Science Ltd Neuropsychologists are frequently faced with the task of judging whether a patient s neuropsychological test scores are indicative of brain dysfunction. One approach to this task, traditionally employed with the Halstead-Reitan battery, is to classify performances as impaired or normal based on conventional raw score cut-offs previously shown to discriminate between brain-damaged and unimpaired subjects (e.g., Halstead, 1947; Russell, Neuringer, & Goldstein, 1970). However, numerous studies have demonstrated that high rates of false-positive impairment are obtained when these conventional cut-scores are employed to classify the performance of healthy older subjects (Bornstein, 1986; Davies, 1968; Elias, Podraza, Pierce, & Robbins, 1990; Heaton, Grant, & Matthews, 1986; Price, Fein, & Feinberg, 1980; Steinmeyer, 1986). Due to the limitations of traditional impairment cut-scores, the more recent trend in neuropsychological interpretation has been to consider the significance of observed scores relative to those obtained among demographically matched healthy normals (Bornstein, 1985; Heaton, Grant, & Matthews, 1991; Spreen & Strauss, 1991). For example, following the classifi- A preliminary version of this report was presented as a poster at the 1995 annual meeting of the National Academy of Neuropsychology, San Francisco, CA. The study was partially supported by a grant from the National Institute of Mental Health (MH43960), and was completed while the first author was a postdoctoral fellow at Harbor-UCLA Medical Center, Torrance, CA. Address correspondence to: Barton W. Palmer, PhD, Geriatric Psychiatry CRC, San Diego VA Medical Center, 3350 La Jolla Village Drive 116A-1, San Diego, CA

2 504 B. W. Palmer et al. cation scheme for Wechsler IQ scores (Wechsler, 1981), test scores falling at least 1.3 standard deviations (SD) below the normative mean are commonly described as borderline, whereas those at least 2.0 SD below the mean may be described as impaired (cf. Lezak, 1995; Orsini, Van Gorp, & Boone, 1988). Together the terms borderline and impaired appear to cover the range implied by another commonly employed phrase, below normal limits. Note, however, that even though the terms can be associated with specific deviation score ranges, an important interpretive question remains: How frequently do healthy normal people obtain scores below normal limits? Heaton et al. (1991) found that the majority of subjects in their sample of neurologically normal adults administered an expanded Halstead-Reitan battery obtained at least one abnormal test score, suggesting that isolated impaired scores may have little interpretive value. While virtually all prior reports of false-positive impairment rates among normals have been presented by investigators using part, all, or an expanded version of the Halstead-Reitan battery (Bornstein, 1986; Davies, 1968; Elias et al., 1990; Heaton et al., 1986, 1991; Price et al., 1980; Steinmeyer, 1986), the issue of false-positive impairment rates may be of even greater concern to neuropsychologists employing a flexible-battery (or hypothesis testing ; Orsini et al., 1988) approach. Specifically, in the flexible-battery approach to neuropsychological assessment there is less reliance on global summary scores (e.g., Average Impairment Rating or Halstead Impairment Index) than is possible with the Halstead-Reitan battery, so individual measures (or small sets of measures grouped by cognitive domain) may carry greater interpretive weight. Clearly then, information is needed about the base rates of falsepositive impairment on such measures and across typical groupings of such measures. The current study was designed to address three basic questions. First, for each of several neuropsychological measures commonly employed in the flexible-battery approach, what proportion of healthy older subjects earn borderline or impaired scores? Second, what proportion of such subjects earn one or more subnormal scores across a typical core battery of such tests? Third, what proportion of such subjects obtain consistently poor performance across multiple measures grouped by cognitive domain? Due to the nature of our available data, the present investigation of these questions was limited to older subjects. Nonetheless, it may be of particular importance to acquire information about the base rates of low test performance in this age group since normal variability in ability appears to increase with age (Johansson, 1990). METHOD Subjects Subjects were 132 healthy normal adults, ages 50 through 79 years (mean (M) 63.8, SD 7.7), whose neuropsychological data was initially collected as a normal comparison group for ongoing research on the cognitive correlates of aging and depression (e.g., Boone et al., 1994, 1995; Palmer et al., 1996). Potential subjects were recruited through newspaper advertisements, and were extensively screened for any risk factors that might adversely impact neurocognitive functioning. Screening procedures included a comprehensive neurological/physical examination and interview regarding medical history, a complete blood chemistry panel, complete blood cell count, and thyroid function panel. Subjects were excluded if there was a history or physical findings suggesting a neurologic disorder, such as dementia, stroke, Parkinson s disease, seizure disorder, or head injury with a loss of consciousness greater than 24 hours. Individuals with laboratory findings showing serious metabolic abnormalities (e.g., low sodium level, elevated glucose level, or thyroid and liver function abnormalities) were also excluded. In

3 Base Rates of Impaired Test Performance 505 addition, potential subjects were interviewed with the Structured Clinical Interview for the DSM-III-R (SCID-nonpatient version; Spitzer & Williams, 1986) to rule out the presence of psychiatric disorders. Psychiatric exclusion criteria were a current or past history of psychotic or mood disorders, substance abuse, or an organic mental disorder. The final sample included 45 males and 87 females. The ethnic makeup of the sample was predominantly Caucasian (117 Caucasians, 9 African Americans, 4 Asian Americans, 2 Hispanics/Latinos). All subjects had at least 12 years of education (M 14.8, SD 2.3). Neuropsychological Assessment Procedures The test battery included many of the measures commonly employed in flexible-battery evaluations (cf. Lezak, 1995; Orsini et al., 1988). Specific cognitive domains and measures, as well as references supporting the use of the various tests to measure these domains, are listed in Table 1. All subjects were tested individually, generally in a single 2 1/2-hour session. Regular rest breaks were given as necessary to avoid subject fatigue. Each test score from each subject was classified as normal, borderline, or impaired, based on the degree to which it deviated from the mean of all similarly aged subjects in the sample (further details of the age groupings and classification of neuropsychological scores are provided in the Results section, below). RESULTS Sample Subgroups Since worse performance on most measures tended to be at least moderately correlated with age (mean absolute magnitude of Pearson r.266), the sample was divided by age decade: (a) 50 to 59 years, (b) 60 to 69 years, and (c) 70 to 79 years. T-tests conducted by gender on all variables were generally nonsignificant, except for immediate recall on the Wechsler Memory Scale-Revised (WMS-R) Logical Memory (LM), t(62) 2.06, p.044, and Categories completed on the Wisconsin Card Sorting Test (WCST), t(127) 2.16, p.032. Neither of the latter remained significant at the α.05 level when multiple comparisons were corrected for by the Bonferroni method. Since the effects of gender appeared less robust than those of age, and since further division of the sample would have yielded relatively small subgroup sizes, the sample was not further divided into age-by-gender subgroups. Similarly, no subdivision of subjects by education were made, however, all subjects had at least a high school education. Frequency of Borderline and Impaired Scores for Each Individual Measure The mean and standard deviation was calculated for each measure for each of the three age groups, as shown in Table 2. In addition, Table 2 shows the proportion of subjects earning scores of at least 1.3 or 2.0 SDs worse than their own age-group mean (borderline and impaired ranges, respectively). 1 1 We considered interpreting subjects test scores relative to published normative data (e.g., Bornstein, 1985; Heaton et al., 1991; Spreen & Strauss, 1991), but ultimately rejected this strategy for two reasons. First, for many of the measures (e.g., Auditory Consonant Trigrams, Stroop, Rey-Osterrieth Figure, Controlled Oral Word Generation) there are no universally accepted ( gold standard ) normative data for older age groups. Second, any other normative sample is likely to have some demographic differences relative to the present sample, in which case deficient performances might be attributable to those minor differences. Nonetheless, based on the means and standard deviations provided in Table 2, interested readers can readily compare the levels of performance leading to classifications of borderline and impaired to the levels that would be similarly classified with any other available normative data.

4 506 B. W. Palmer et al. Domains and Measures TABLE 1 Cognitive Domains and Measures Cross cognitive functioning Mini-Mental State Exam (MMSE; Folstein, Folstein, & McHugh, 1975) Basic attention Digit Span-scaled score (Wechsler, 1981) Mental processing speed Digit Symbol scaled score (Wechsler, 1981) Stroop Words & Colors seconds (Goodglass & Kaplan, 1979; Lezak, 1995) Language Boston Naming Test (BNT; Kaplan, Goodglass, & Weintraub, 1983) Controlled Oral Word Generation Test total words (FAS; Lezak, 1995) Visual-construction Key-Osterrieth Complex Figure copy score (Lezak, 1995) Verbal memory Logical Memory (LM) subtest from Wechsler Memory Scale (WMS; Wechsler, 1945; Russell, 1975) or Wechsler Memory Scale-Revised (WMS-R; Wechsler, 1987): Immediate recall (LM I) 30-minute delayed recall Percent retention Recognition Memory Test total words (RMT-Words; Warrington, 1984) Visual memory Visual Reproductions (VR) subtest from WMS (Wechsler, 1945) or WMS-R (Wechsler, 1987; Russell, 1975): Immediate recall (VR I) 30-minute delayed recall Percent retention Rey-Osterrieth Complex Figure (Lezak, 1995) 3-minute delayed recall (3 -delay) Percent retention Recognition Memory Test total faces (RMT-Faces; Warrington, 1984) Frontal/executive skills Controlled Oral Word Generation Test total words (FAS; Lezak, 1995) Stroop Interference seconds (Goodglass & Kaplan, 1979; Lezak, 1995) Auditory Consonant Trigrams (ACT; Lezak, 1995; Stuss et al., 1982) total consonants correctly recalled Wisconsin Card Sorting Test a (WCST; Heaton, Chelune, Talley, Kay, & Curtiss, 1993): Categories Errors Perseverative Responses Percent perseverative errors Failure to maintain set Other (i.e., responses not matching color, form, or number) Percent conceptual responses Trials to first category a The WCST protocols were computer scored to eliminate concerns about interrater reliability (Harris, 1988). As can be seen in Table 2, each measure yielded at least some proportion of subjects in the borderline range (in some cases exceeding the 8% expected from a normal distribution). Furthermore, most measures yielded at least some proportion of subjects in the impaired range, the only exceptions being Digit Span, Digit Symbol, and FAS (many exceeding the 2% expected from a normal distribution). Frequency of Borderline and Impaired Range Scores Across the Battery Among Individual Subjects For each subject, we calculated the total number of borderline or impaired scores earned across all measures. As seen in Table 3, 73% (97/132) of subjects earned at least one score in the borderline range, and 48% (63/132) earned at least two scores in the borderline range.

5 TABLE 2 Proportion of Healthy Subjects Obtaining Borderline ( 1.3 SD) or Impaired ( 2.0 SD) Scores on Individual Neuropsychological Measures Age (n 43) Age (n 53) Age (n 36) Measure M (SD) % 1.3 % 2.0 M (SD) % 1.3 % 2.0 M (SD) % 1.3 % 2.0 MMSE 29.5 (0.8) (0.8) (1.4) Digit Span 11.0 (2.1) (2.4) (2.3) Digit Symbol 9.4 (2.4) (2.3) (1.9) BNT 56.8 (2.7) (3.6) (9.1) FAS 41.6 (7.7) (12.5) (12.0) WMS-R a LM I 25.8 (6.2) (6.4) (7.9) LM % retention 82.7 (12.3) (16.2) (21.2) VR I 35.8 (4.2) (5.2) (7.6) VR % retention 79.3 (16.8) (23.9) (30.3) RMT-Words 47.2 (3.4) (2.6) (6.7) RMT-Faces 44.2 (3.7) (4.2) (6.6) Rey-Osterrieth Figure Copy 34.4 (2.0) (2.6) (2.8) minute delay 19.1 (5.1) (6.2) (5.6) Percent retention 55.2 (14.3) (16.7) (16.6) Stroop Words 41.7 (6.5) (8.5) (8.4) Colors 58.1 (10.3) (11.4) (16.6) Interference (33.2) (31.4) (55.5) ACT 47.9 (6.9) (7.5) (9.5) WCST Categories 5.0 (1.7) (1.7) (2.0) Errors 27.6 (20.8) (23.3) (23.0) Perseverative responses 15.0 (13.6) (23.2) (24.8) Percent perseverative errors 12.9 (7.8) (13.1) (13.9) Failure to maintain set 0.7 (1.1) (1.2) (1.3) Other 1.0 (2.4) (2.9) (2.3) Percent conceptual responses 66.4 (19.5) (21.1) (23.2) Trials to first category 13.6 (4.9) (16.8) (34.5) ACT Auditory Consonant Trigrams; BNT Boston Naming Test; FAS Controlled Oral Word Generation Test; LM Logical Memory; M mean; MMSE Mini-Mental State Exam; RMT Recognition Memory Test; SD standard deviation; VR Visual reproductions; WMS-R Wechsler Memory Scale-Revised; WSCT Wisconsin Card Sorting Test. Percent Borderline or Impaired (% 1.3), percent Impaired (% 2.0). a The Russell (1975) revision of the LM and VR subtests from the original WMS was administered to approximately half the subjects rather than the parallel subtests from the WMS- R. The values provided for LM and VR in this table were derived solely from subjects receiving the WMS-R, however, results with the WMS were generally consistent with those for the WMS-R. 507

6 508 B. W. Palmer et al. TABLE 3 Percent Healthy Normal Subjects Obtaining One or More Borderline or Impaired Scores Over the Entire Battery Number of Scores or Tests All scores Borderline or worse Impaired Different tests a Borderline or worse Impaired a Different tests multiple borderline or impaired scores from any single test counted only once. When using the more stringent cut-off of 2.0 SDs below the mean of age matched peers, 37% (49/132) of subjects earned at least one impaired range score, and 24% (31/132) earned at least two impaired range scores. It is possible that the figures from the preceding analyses were unduly inflated by the multiple scores from some tasks (e.g., WCST). Thus, as shown in Table 3, we further calculated the number of subjects earning borderline or impaired scores on at least two different tests. 2 Of the 132 subjects, 44% (58/132) earned at least two scores in the borderline or lower range from two distinct tests. Furthermore, 20% (26/132) earned at least two scores in the impaired range on at least two different tests. Consistency of Abnormal Scores Within Domains Among Individual Subjects Greater interpretative weight is typically given when a pattern of similarly deficient performance is observed on multiple neuropsychological measures within a given cognitive domain. We lacked multiple measures in some commonly assessed domains (e.g., basic attention and visual-construction). On the other hand, there were several scores typically classified as memory measures : LM, Visual Reproductions (VR), Rey-Osterrieth (3-minute delay or percent retention), Recognition Memory Test (RMT) Words and Faces. These memory measures may also be considered in terms of the subdomains of verbal memory (LM and RMT- Words), and visual memory (VR, Rey-Osterrieth, and RMT-Faces). Several tests that may be associated with frontal/executive functioning were also available: FAS, Stroop Interference, Auditory Consonant Trigrams, and WCST. In addition, we had three measures sensitive to mental processing speed: Digit Symbol, Stroop Words, and Stroop Colors. There were also two measures related to language functioning: FAS and Boston Naming Test (BNT). Using subjects within each age group as their own normative sample, the number of subjects earning borderline or impaired scores on multiple measures within each of these domains was calculated, as shown in Table 4. In general, the base rates of multiple borderline or impaired performances within domains appears substantially lower than the figures in the previous analyses. 2 Multiple scores that were derived from the same performance (e.g., Rey-Osterrieth Figure: 3-minute delay and percent retention) were counted only once. However, since the RMT-Words and RMT-Faces, and the three Stroop test scores, were each derived from separate task performances, these latter scores were counted as independent tests in analyzing the consistency of subjects performance within cognitive domains.

7 Base Rates of Impaired Test Performance 509 TABLE 4 Percent Healthy Normal Subjects Scoring in the Borderline or Impaired Ranges on Different Tests Within Specific Cognitive Domains Number of Tests Mental processing speed Borderline or worse n/a n/a Impaired n/a n/a Language Borderline or worse n/a n/a n/a Impaired n/a n/a n/a Memory Borderline or worse Impaired Verbal memory Borderline or worse n/a n/a n/a Impaired n/a n/a n/a Visual memory Borderline or worse n/a n/a Impaired n/a n/a Frontal/executive Borderline or worse n/a Impaired n/a Note. Cognitive domains defined as follows: Mental processing speed: Digital Symbol, Stroop Words, and Stroop Colors; Language: Boston Naming Test (BNT) and Controlled Oral Word Generation Test (FAS); Memory: Logical Memory (LM), Visual Reproductions (VR), Rey-Osterrieth (3-minute delay or percent retention), Recognition Memory Test (RMT) Words and RMT-Faces; Verbal memory: LM and RMT-Faces; Visual memory: VR, Rey-Osterrieth (3-minute delay or percent retention), and RMT-Faces; Frontal/Executive: FAS, Stroop Interference, Auditory Consonant Trigrams (ACT) and Wisconsin Card Sorting Test (WCST). DISCUSSION The above findings reveal that on each of several neuropsychological measures often employed in flexible neuropsychological batteries, at least a small proportion of healthy older adults earn borderline or impaired scores. Across a typical battery of such tests, the majority of our subjects (73%) earned at least one borderline score, over one third (37%) earned at least one impaired range score, and one fifth earned at least two impaired range scores on separate tests. Relatively few subjects evidenced consistently impaired performance across multiple measures within specific cognitive domains. The generalizability of the current findings may be partially constrained by the demographic nature of our sample, that is, subjects were older, relatively well educated, and predominantly Caucasian. It is not fully clear how the results might have differed with a sample with vastly different demographic characteristics. For example, although better performance on many neuropsychological measures tends to be positively associated with educational level (Lezak, 1995), there may also be a wider degree of normal variability in the levels of neurocognitive ability within less educated samples (cf. Welch, Doineau, Johnson, & King, 1996). If interpreted relative to the means and standard deviations of education-matched peers, individuals with little education might need to obtain a much lower raw score than would highly educated subjects in order to be classified within the borderline or impaired ranges. Thus, further research may be needed to determine the degree to which the present pattern of findings generalizes to individuals from markedly different demographic backgrounds. The present study may also be limited by the relatively small subgroup sample sizes, particularly in the oldest age group. A larger sample would have obvious advantages in terms

8 510 B. W. Palmer et al. of accurately determining normative values, and would also permit further subdivision of subjects by other potentially important demographic factors such as, gender and education (cf. Heaton et al., 1991). Another issue deserving comment is our implicit assumption that borderline or impaired scores among normals represented false-positive findings. It is conceivable that some of the deficient neuropsychological performances observed among our subjects reflected genuine, albeit subtle, neurocognitive impairments. On the other hand, all subjects in the present study were extensively screened for any factors that might have adversely impacted neurocognitive functioning, and in no case did they report any history suggestive of impairment in daily functional abilities. Thus, it seems highly unlikely that a significant proportion of these subjects had clinically relevant degrees of genuine neurocognitive deficits. Despite the above limitations, the present findings have some clear clinical implications. Paralleling prior reports using the Halstead-Reitan battery (Bornstein, 1986; Davies, 1968; Elias et al., 1990; Heaton et al., 1986, 1991; Price et al., 1980; Steinmeyer, 1986), our results demonstrate that even well-screened healthy older normal subjects frequently obtain isolated abnormal scores. It is possible that even higher rates of false-positive impairment would have been observed among normals having significant nonneurological medical illnesses. Within the context of civil litigation, neuropsychologists are frequently asked to render an expert opinion regarding the likelihood that a given incident caused significant decrements in a patient-plaintiff s neurocognitive abilities. In light of the present findings, any strong assertions regarding the occurrence of neurocognitive injury appear difficult to justify solely on the basis of a few unrelated and isolated abnormal test scores. The present findings also have important implications for general clinical neuropsychology practice. In the current climate of managed health care, some neuropsychologists may find themselves pressured by time and economic factors to make clinical judgments based on short screening batteries. While findings of impairment on isolated neuropsychological scores may be useful for developing tentative hypotheses, the present findings illustrate the need for clinicians to actively seek data that might disconfirm such hypotheses by administering multiple measures and considering the degree of consistency within cognitive domains. While few competent clinicians are likely to interpret test data without taking additional information into account, sensitivity to the base rates of deficient performance among normals on individual tests and groups of tests is indispensable to the extent that neuropsychological test scores are given any weight in the diagnostic process. REFERENCES Boone, K. B., Lesser, I. M., Miller, B. L., Wohl, M., Berman, N., Lee, A., & Palmer, B. (1994). Cognitive functioning in a mildly to moderately depressed geriatric sample: Relationship to chronological age. Journal of Neuropsychiatry and Clinical Neurosciences, 6, Boone, K. B., Lesser, I. M., Miller, B. L., Wohl, M., Berman, N., Lee, A., Palmer, B, & Back, C. (1995). Cognitive functioning in older depressed outpatients: Relationship of presence and severity of depression to neuropsychological test scores. Neuropsychology, 9, Bornstein, R. A. (1985). Normative data on selected neuropsychological measures from a nonclinical sample. Journal of Clinical Psychology, 41, Bornstein, R. A. (1986). Classification rates obtained with standard cut-off scores on selected neuropsychological measures. Journal of Clinical and Experimental Neuropsychology, 8, Davies, A. D. M.. (1968). The influence of age on trail making test performance. Journal of Clinical Psychology, 24, Elias, M. F., Podraza, A. M., Pierce, T. W., & Robbins, M. A. (1990). Determining neuropsychological cut scores for older, healthy adults. Experimental Aging Research, 16,

9 Base Rates of Impaired Test Performance 511 Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). Mini-Mental State. Journal of Psychiatric Research, 12, Goodglass, H., & Kaplan, E. (1979). Assessment of cognitive deficit in the brain-injured patient. In M. S. Gazzaniga (Ed.), Handbook of behavioral neurobiology: Vol. 2. Neuropsychology (pp. 3 22). New York: Plenum Press. Halstead, W. C. (1947). Brain and intelligence: A qualitative study of the frontal lobes. Chicago: University of Chicago Press. Harris, M. E. (1988). Wisconsin Card Sorting Test: Computer version. Odessa, FL: Psychological Assessment Resources. Heaton, R. K., Chelune, G. J., Talley, J. L., Kay, G. G., & Curtiss, G. (1993). Wisconsin card sorting test manual: Revised and expanded. Odessa, FL: Psychological Assessment Resources. Heaton, R. K., Grant, I., & Matthews, C. G. (1986). Differences in neuropsychological test performance associated with age, education, and sex. In I. Grant & K. M. Adams (Eds.), Neuropsychological assessment of neuropsychiatric disorders (pp ). New York: Oxford University Press. Heaton, R. K., Grant, I., & Matthews, C. G. (1991). Comprehensive norms for an expanded Halstead-Reitan battery: Demographic corrections, research findings, and clinical applications. Odessa, FL: Psychological Assessment Resources. Johansson, B. (1990). Neuropsychological assessment in the oldest-old. International Psychogeriatrics, 3(Suppl.), Kaplan, E. F., Goodglass, H., & Weintraub, S. (1983). The Boston naming test (2nd ed.). Philadelphia: Lea & Febiger. Lezak, M. D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press. Orsini, D. L., Van Gorp, W. G., & Boone, K. B. (1988). The neuropsychology casebook. New York: Springer- Verlag. Palmer, B. W., Boone, K. B., Lesser, I. M., Wohl, M. A., Berman, N., & Miller, B. L. (1996). Neuropsychological deficits among older depressed patients with predominantly psychological or vegetative symptoms. Journal of Affective Disorders, 41, Price, L. J., Fein, G., & Feinberg, I. (1980). Neuropsychological assessment of cognitive function in the elderly. In L. W. Poon (Ed.). Aging in the 1980s: Psychological issues (pp ). Washington, DC: American Psychological Association. Russell, E. W. (1975). A multiple scoring method for the assessment of complex memory functions. Journal of Consulting and Clinical Psychology, 43, Russell, E. W., Neuringer, C., & Goldstein, G. (1970). Assessment of brain damage: A neuropsychological key approach. New York: John Wiley & Sons. Spitzer, R. L., & Williams, J. B. (1986). Structured clinical interview for the DSM-III-R. New York: New York State Psychiatric Institute. Spreen, O., & Strauss, E. (1991). A compendium of neuropsychological tests: Administration, norms, and commentary. New York: Oxford University Press. Steinmeyer, C. H. (1986). A meta-analysis of Halstead-Reitan test performances of non-brain damaged subjects. Archives of Clinical Neuropsychology, 1, Stuss, D. T., Kaplan, E. F., Benson, D. F., Weir, W. S., Chiulla, S., & Sarazin, F. F. (1982). Evidence of the involvement of orbitofrontal cortex in memory functions: An interference effect. Journal of Comparative and Physiological Psychology, 96, Warrington, E. K. (1984). Recognition Memory Test. Manual. Berkshire, England: Nfer-Nelson. Wechsler, D. (1945). A standardized memory scale for clinical use. Journal of Psychology, 19, Wechsler, D. (1981). Wechsler Adult Intelligence Scale Revised manual. New York: Psychological Corporation. Wechsler, D. (1987). Wechsler Memory Scale Revised. New York: Psychological Corporation. Welch, L. W., Doineau, D., Johnson, S., & King, D. (1996). Education and gender normative data for the Boston Naming Test in a group of older adults. Brain and Language, 53,