Use of the WCST and the WCST-64 in the Assessment of Traumatic Brain Injury

Transcription

1 Journal of Clinical and Experimental Neuropsychology /03/ $ , Vol. 25, No. 4, pp # Swets & Zeitlinger Use of the WCST and the WCST-64 in the Assessment of Traumatic Brain Injury Mark Sherer 1,2, Todd G. Nick 2, Scott R. Millis 3,4, and Thomas A. Novack 5,6 1 Methodist Rehabilitation Center, Jackson, MS, USA, 2 University of Mississippi Medical Center, Jackson, MS, USA, 3 Kessler Medical Rehabilitation Research and Education Corporation, West Orange, NJ, USA, 4 University of Medicine & Dentistry of New Jersey New Jersey Medical School, West Orange, NJ, USA, 5 Spain Rehabilitation Center, Birmingham, AL, USA, and 6 University of Alabama-Birmingham, Birmingham, AL, USA ABSTRACT The Wisconsin Card Sorting Test (WCST) has been found to be valid in characterizing cognitive dysfunction in a variety of neurological disorders including traumatic brain injury (TBI). However, the test has been criticized as being too lengthy and frustrating for severely impaired patients. As a result, shortened versions of the WCST have been proposed. The utility of one of these shortened versions, the Wisconsin Card Sorting Test-64 (WCST-64), was examined in 106 persons with TBI. Findings showed strong associations between scores derived from the two tests. WCST scores were predicted from WCST-64 scores with sufficient accuracy for research purposes with adjusted R-squared values ranging from.74 to.87. Using the standard cutpoint of < 40T to indicate impairment or normal performance for perseverative responses from each of the two tests, 91 (86%) of 106 subjects received the same classification showing substantial agreement (Kappa statistic 0.71; 95% CI ). The WCST and the WCST-64 also performed comparably in predicting functional status at discharge from inpatient rehabilitation using the first score from a principal components analysis as a summary measure (both significant with p¼.0002). These findings support the use of the WCST-64 in early evaluations of persons with moderate and severe TBI. The Wisconsin Card Sorting Test (WCST) is a measure of abstract reasoning ability that requires the ability to shift cognitive strategies (Heaton, Chelune, Talley, Kay, & Curtiss, 1993). The WCST is considered to be a measure of executive functions because successful performance requires planning, utilization of feedback, cognitive flexibility, goal directed behavior, and ability to inhibit impulsivity (Heaton et al., 1993). The WCST has been used in the assessment of a variety of neurological disorders including seizure disorders, multiple sclerosis, Parkinson s disease, Korsakoff s syndrome, schizophrenia, and traumatic brain injury (TBI). Some investigators have found the WCST to be selectively sensitive to frontal lobe lesions (Drewe, 1974; Milner, 1964; Robinson, Heaton, Lehman, & Stilson, 1980) while others (Anderson, Damasio, Jones, & Tranel, 1991) have failed to replicate these findings. Given the common occurrence of frontal lobe lesions (Graham, 1996) and impaired executive functions (Sherer, Madison, & Hannay, 2000) in persons with TBI, the WCST would appear to be a particularly appropriate instrument for this patient population. The WCST is frequently used in clinical trials of patients with TBI (Clifton, Hayes, Levin, Michel, & Choi, 1992; Hannay & Sherer, 1996). Previous investigations have shown that the WCST is sensitive to Address correspondence to: Mark Sherer, Ph.D., ABPP-Cn, Neuropsychology Department, Methodist Rehabilitation Center, 1350 E. Woodrow Wilson, Jackson, MS 39216, USA. Tel.: þ marks@mmrcrehab.org Accepted for publication: September 5, 2002.

2 WCST AND WCST neurobehavioral impairments after TBI (Ferland, Ramsay, Engeland, & O Hara, 1998; Kreutzer, Gordon, Rosenthal, & Marwitz, 1993; Stuss, 1987). Other investigations have shown that the WCST is predictive of risk for falls (Rapport, Hanks, Millis, & Despande, 1998), long-term employment outcome (Boake et al., 2001; Najenson, Grosswasser, Mendelson, & Hackett, 1980), degree of impaired self-awareness (Trudel, Tryon, & Purdum, 1998), and level of independence (Little, Templer, Persel, & Ashley, 1996) after TBI. Despite the proven clinical utility of the WCST, the test has been criticized for being too lengthy and too frustrating for severely impaired patients (Robinson, Kester, Saykin, Kaplan, & Gur, 1991; Smith-Seemiller, Franzen, & Bowers, 1997). Consequently, there has been interest in developing a shorter, more user-friendly version of the test. Although several modifications of the WCST have been examined, the Wisconsin Card Sorting Test-64 (WCST-64) has received the most empirical support (Axelrod, Henry, & Woodard, 1992; Paolo, Axelrod, Troster, Blackwell, & Koller, 1996; Robinson et al., 1991; Smith- Seemiller et al., 1997). Axelrod, Paolo, and Abraham (1997) noted that simple conversation of WCST-64 scores to percentages of cards attempted (always 64) and use of WCST norms is not appropriate. These authors called for more sophisticated methods for estimating WCST scores from the WCST-64 such as regression modeling or norms developed especially for the WCST-64. Separate norms for the WCST-64 have now been published (Kongs, Thompson, Iverson, & Heaton, 2000). Early review of the WCST-64 norm set and accompanying manual (Greve, 2001) suggests that this test may be interpreted similarly to the WCST, but additional research is clearly needed. The WCST-64 is administered similarly to the WCST, but only 64 cards are presented rather than all 128 cards or a sufficient number of cards to reach the criterion of six categories achieved as with the regular WCST (Kongs et al., 2000). Similar scores can be calculated for the WCST-64 as for the WCST. As noted above, comprehensive norms for the WCST-64 are provided by Kongs et al. (2000). To date, there has been only limited investigation of use of the WCST-64 with persons with TBI. In a study of a mixed neurologic and psychiatric sample with a substantial percentage (47%) of patients with TBI, Smith-Seemiller et al. (1997) found that WCST-64 scores were highly correlated with WCST scores. Using a simple cutoff score on perseverative responses, 86% of patients were received the same classification (impaired vs. not impaired) with the WCST-64 as with the WCST. Iverson, Slick, and Franzen (2000) provided clinical norms from a sample of 303 patients with uncomplicated mild TBI. No comparisons to controls or patients with more severe TBI were provided. The current investigation examined the interrelationships of WCST-64 and WCST scores in a sample of persons in early recovery from TBI. We developed regression equations for estimating WCST scores from WCST-64 scores and to determine the percentage of shared variability. We also examined agreement between the two tests in classifying subject performances as impaired versus normal. Finally, we compared the abilities of the WCST-64 and the WCST to predict level of disability at discharge from inpatient rehabilitation. METHOD Study Population The study population consisted of qualified persons with TBI who were admitted to 1 of 2 inpatient brain injury rehabilitation programs (Methodist Rehabilitation Center and Spain Rehabilitation Center). The majority of participants were recruited as part of National Institute on Disability and Rehabilitation Research (NIDRR) TBI Model Systems programs at the two sites. Inclusion criteria for the TBI Model Systems program include: medically documented TBI; treatment at an affiliated Level I trauma center within 24 hr of injury; receipt of inpatient rehabilitation within the Model System; admission to inpatient rehabilitation within 72 hr of discharge from acute care; age of at least 16 at the time of injury; and provision of informed consent by the person with injury or a legal proxy. Nonsystem subjects were similar to TBI Model Systems subjects except that they received emergency and acute medical care at hospitals other than the affiliated Level 1 trauma centers. To qualify for the

3 514 MARK SHERER ET AL. present study, subjects were also required to have emerged from posttraumatic amnesia (PTA) prior to discharge from inpatient rehabilitation and to be able to complete a comprehensive neuropsychological evaluation in English. Resolution of PTA was defined as two consecutive scores of greater than 75 on the Galveston Orientation and Amnesia Test (GOAT; Levin, O Donnell, & Grossman, 1979). Data Collection Demographic information (age, gender, years of education), and injury severity data were collected through review of medical records and interview with patients and family members. Glasgow Coma Scale (GCS) scores were obtained at admission to the Emergency Departments following TBI. Severity was classified in the usual way with scores from 3 to 8 indicating severe TBI, scores from 9 to 12 indicating moderate TBI, and scores from 13 to 15 indicating mild TBI. Chronicity of injury was calculated as the interval in days from date of injury to date of the initial neuropsychological assessment. Comprehensive neuropsychological assessments including the WCST were completed on all subjects at resolution of PTA or shortly after admission for inpatient rehabilitation for patients whose PTA resolved prior to rehabilitation admission. WCST and WCST-64 scores used for this investigation were collected in the following manner. All subjects were administered the full WCST in accordance with standard procedures. All responses were scored using Wisconsin Card Sorting Test: Computer Version 2 Research Edition software purchased from Psychological Assessment Resources, Inc. (PAR) to obtain the full WCST scores. The first 64 responses were then re-scored using WCST-64: Computer Version for Windows Research Edition software also purchased from PAR to obtain the WCST-64 scores. Disability Rating Scale data were collected at discharge from inpatient rehabilitation. WCST and WCST-64 scores selected for analysis in the present investigation were perseverative responses, perseverative errors, nonperseverative errors, categories completed, and conceptual level responses. These scores were selected based on review of previous WCST articles, consultation with authorities on the WCST, and the likelihood of meaningful prediction of WCST scores from WCST-64 scores. For example, failure to maintain set was fairly infrequent for both the WCST and the WCST-64 in our sample and had a very limited range. Prediction of this score would likely be quite poor. On the other hand, trials to first category is the same for the WCST and the WCST-64 so no prediction formula is needed. The Disability Rating Scale (DRS, Rappaport, Hall, Hopkins, Belleza, & Cope, 1982) is a 30-point scale which rates eight areas of functioning: eye opening; verbalization; motor response; level of cognitive ability for daily activities of feeding, toileting, and grooming; overall level of dependence; and employability. Each area of functioning is rated on a scale of 0 to 3, 4, or 5, with higher scores representing lower levels of functioning. Scores on each item are summed to yield a total score between 0 and 30, with a higher score indicating greater disability. The DRS has been shown to be a valid and sensitive measure of outcome after TBI (Hall, Cope, & Rappaport, 1985). Interrater reliability for the DRS has ranged from 0.97 to 0.98 in previous investigations (Gouvier, Blanton, LaPorte, & Nepomuceno, 1987; Rappaport et al., 1982). DRS ratings were made independently of collection of WCST data by the treating physician. Statistical Analyses Spearman correlation coefficients were calculated to examine bivariable associations between WCST scores and the corresponding WCST-64 scores. Regression models for estimating WCST scores from WCST-64 scores were developed using standard multiple regression techniques allowing for nonlinear effects. Predictors examined for a given WCST score included age, education, and the corresponding WCST-64 score. Age and education were included as existing norms for both tests include adjustments for age and education. It was possible that differential age and education effects might be observed for the two tests. Four models were developed for each score. These were models with only the corresponding WCST-64 score assuming linear effects and allowing for nonlinear effects and models with age, education, and the corresponding WCST-64, again, both assuming linear effects and allowing for nonlinear effects. Agreement in classification of impairment or normal performance was calculated for the two tests. The score selected for comparison was perseverative responses based on prior research showing that this score is the most sensitive to brain impairment (Heaton, Grant, & Matthews, 1991). The recommended cut-off of <40T was used to determine impaired vs. normal performance (Heaton et al., 1993; Kongs et al., 2000). Previous investigations have shown high intercorrelations among WCST scores and among WCST-64 scores. WCST scores and WCST-64 scores were submitted to separate principal component analyses in order to reduce the numbers of variables entered into multiple regression analyses to predict DRS at rehabilitation discharge and to avoid problems with the regression models caused by intercorrelations of predictor variables. Multiple regression analyses were then performed using age, education, GCS, chronicity,

4 WCST AND WCST and either the WCST component scores or the WCST-64 component scores to predict DRS at rehabilitation discharge. Comparison of these two analyses permitted determination of the relative validities of the WCST and the WCST-64 for predicting this outcome. RESULTS During the study period, 106 qualified subjects were admitted to the two centers. Of these, 32 were TBI Model Systems subjects admitted to Spain Rehabilitation Center, 28 were TBI Model Systems subjects admitted to Methodist Rehabilitation Center, and 46 were non-tbi Model Systems subjects admitted to Methodist Rehabilitation Center. Characteristics of the study sample are presented in Table 1. As with other series of TBI patients, the majority are young males. Table 1. Description of the Study Sample on Demographics, Predictors Other than WCST Scores, and DRS at Rehabilitation Discharge. Categorical descriptors Missing (%) n (%) Sex 0 Male 80 (76%) Female 26 (24%) Race 0 White 81 (76%) African American 25 (24%) GCS Total 1 (1%) (56%) (29%) (15%) Continuous descriptors Missing (%) Median [25th, 75th percentile] Age 0 26 [20, 39] Education 0 12 [11, 13] Chronicity (days) 0 34 [23, 54] DRS total at rehab discharge 0 6 [ 5, 8] Note. GCS ¼Glasgow Coma Scale, DRS ¼Disability Rating Scale. Table 2. Descriptive Statistics and Spearman Correlations for WCST-64 and WCST Scores. Mean (s) 25th/ 50th/75th percentiles Correlations of WCST-64 and WCST Perseverative responses.92 WCST 42.8 (32.6) 16/40/57 WCST (15.2) 9/17/26 Nonperseverative errors.87 WCST 18.4 (14.6) 7/15/25 WCST (7.4) 5/8/13 Perseverative errors.92 WCST 35.1 (24.2) 14/33/48 WCST (11.0) 9/14/23 Categories completed.89 WCST 3.4 (2.2) 2/3/6 WCST (1.5) 1/2/3 Conceptual level responses.86 WCST 47.3 (21.5) 33/54/64 WCST (16.1) 18/29/42 Note. All Spearman correlations significant at the.01 level. Means, standard deviations, quartiles, and intercorrelations of WCST-64 and WCST scores for perseverative responses (PR), perseverative errors (PE), nonperseverative errors (NPE), categories completed (CC), and conceptual level responses (CLR) are presented in Table 2. In each case, the WCST-64 score has a strong association with the corresponding WCST score with Spearman correlation coefficients ranging from.86 to.92. Regression models for predicting WCST scores from WCST-64 scores were developed using univariable linear regression of WCST-64 scores, univariable regression of WCST-64 scores with linearity assumptions relaxed, multivariable linear regression of age, education, and WCST-64 scores, and, finally, multivariable regression of age, education, and WCST-64 scores with linearity assumptions relaxed for the WCST-64 scores. A comparison of variability accounted for by these various regression models is presented in Table 3. Inspection of Table 3 indicates that the linear univariable models are the most efficient models for predicting PR, PE, and NPE with

5 516 MARK SHERER ET AL. Table 3. Comparison of Variability Explained (Adjusted R 2 ) by Various Regression Models for Predicting WCST Scores from WCST-64 Scores. Model df PR PE NPE CC CLR WCST WCST linearity relaxed WCST-64, age, education WCST-64 linearity relaxed, age, education Note. df ¼degree of freedom, PR ¼perseverative responses, PE¼perseverative errors, NPE ¼ nonperseverative errors, CC¼ categories completed, CLR ¼conceptual level responses. Linearity assumptions relaxed using restricted cubic splines with five knots. R 2 values shown in boldface indicate the most efficient model for predicting a particular score. adjusted R 2 s of.87,.86, and.84, respectively. Univariable models with linearity assumptions relaxed were the most efficient models for predicting CC and CLR with adjusted R 2 s of.81 and.79, respectively. Age and education did not significantly contribute to prediction. Agreement in classification of subjects performances as impaired (<40T) or normal (40T) was determined. A scatter plot showing agreement and disagreement in classification is shown in Figure 1. The intraclass correlation coefficient (ICC) for the two sets of scores was.79 (95% CI ) indicating substantial agreement (Landis & Koch, 1977). Of the 106 subjects, 91 (86%) received the same classification. Subjects were more likely to be classified as impaired by the WCST as compared to the WCST-64 (65% vs. 53%). For subjects classified as impaired by the WCST, 80% were also classified as impaired by the WCST-64. For subjects classified as impaired by the WCST-64, 98% were classified as impaired by the WCST. For subjects classified as normal by the WCST, 97% were classified as normal by the WCST-64 while for subjects classified as normal by the WCST-64, only 72% were classified as normal by the WCST. The kappa statistic for agreement in classification (impaired Fig. 1. Agreement in classification as impaired vs. normal based on WCST and WCST-64 perseverative responses.

6 WCST AND WCST vs. normal) was 0.71 (95% CI ) indicating substantial agreement (Landis & Koch, 1977). Scores were within 10 T scores of each other for 72% of subjects. Lower (more impaired scores) were obtained by 58% of subjects on the WCST while lower scores were obtained by 32% subjects on the WCST-64 and 10% of subjects receive the same T score on both tests. The median T score difference was 6. Prior to developing regression models for predicting DRS at rehabilitation discharge WCST and WCST-64 scores were submitted to principal components analysis for data reduction and to avoid modeling problems caused by the strong intercorrelations among each set of scores. Results of the two principal components analyses are shown in Table 4. The two analyses resulted in very similar two component solutions with PR, PE, CC, and CLR determining Component 1 and NPE determining Component 2. For the WCST, Component 1 accounted for 70% of variability and Component 2 accounted for 25% of variability for a total of 95%. For the WCST-64, Component 1 accounted for 68% of variability and Component 2 accounted for 29% of variability for a total of 97%. In order to compare the abilities of WCST and WCST-64 to predict DRS at rehabilitation discharge, two multiple regressions models were developed. Each model included age, education, GCS, and chronicity as predictors and the first model also included scores for Components 1 and 2 calculated from WCST scores while the second model included scores for Components 1 and 2 calculated from WCST-64 scores. The two models are shown in Table 5. All effects are interquartile-range coefficients. Interquartilerange coefficients are the effects (changes in predicted DRS score) of increasing each predictor variable from its lower quartile to its upper Table 4. Principal Component Analyses of WCST and WCST-64 Scores. Wisconsin score WCST WCST-64 Component 1 Component 2 Component 1 Component 2 PR PE NPE CC CLR Note. PR¼perseverative responses; PE ¼perseverative errors; NPE ¼ nonperseverative errors; CC¼ categories completed; CLR¼ conceptual level responses. Table 5. Regression Models for Predicting DRS at Rehabilitation Discharge from WCST and WCST-64 Scores. Predictors WCST WCST-64 25th, 75th Effect (95% CI) p value 25th, 75th Effect (95% CI) p value Comparison Comparison Age 20, ( 0.79, 0.41).53 20, ( 0.65, 0.53).84 Education 11, ( 0.23, 0.47).50 11, ( 0.20, 0.50).41 GCS 6, ( 0.50, 0.66).79 6, ( 0.58, 0.60).98 Chronicity 23, (0.77, 2.19) , (0.80, 2.23).0002 Component , (0.47, 1.95) , (0.37, 1.60).002 Component , ( 0.70, 0.05) , ( 0.64, 0.09).15 Note. CI¼ confidence interval. All effects are interquartile-range coefficients. Interquartile-range coefficients are the effects (changes in predicted DRS score) of increasing each predictor variable from its lower quartile to its upper quartile.

7 518 MARK SHERER ET AL. quartile. Chronicity was a significant predictor of DRS in each model. Participants evaluated at later times postinjury had greater levels of disability at rehabilitation discharge. Since patients were tested at resolution of PTA, patients evaluated at later times postinjury had sustained more severe injuries and were expected to have greater functional impairment. Component 1 was significant in each model with similar effects while Component 2 was significant in the WCST model with only a trend for an effect in the WCST-64 model. Patients with better performances on WCST or WCST-64 at initial neuropsychological evaluation showed lower levels of disability at rehabilitation discharge. Adjusted R 2 for the WCST model was.24 while adjusted R 2 for the WCST-64 model was.22. DISCUSSION Consistent with prior investigations, we found that WCST-64 scores showed strong associations with corresponding WCST scores. Regression models for estimating WCST scores from WCST-64 scores accounted for 79 87% of the variability in the predicted scores after adjustment for shrinkage. These findings indicate that WCST scores can be predicted with good accuracy using scores from the WCST-64. We believe that these regression formulae will be useful for estimating WCST scores from WCST-64 scores for research purposes. Researchers with existing WCST data bases could switch to the WCST-64 and estimate WCST scores in this manner. This method of estimating WCST scores is superior to the percentage conversion method described by Axelrod et al. (1997). Nonetheless, we do not recommend use of these predictions for clinical purposes unless additional investigation shows this to be a clinically valid procedure. In our view, it is more appropriate to explore the clinical validity of WCST-64 scores directly rather than based on their relationships to WCST scores. Our investigation of agreement in classification of impairment between the two tests showed substantial agreement. The WCST appears to be more sensitive to impairment than the WCST-64 as a higher percentage of subjects produced impaired scores on the WCST. Some subjects appeared to breakdown and show greater impairment on the second deck of cards. The apparent greater sensitivity of the WCST indicates that it may be the more appropriate test for patients who are tested at a later time postinjury when their deficits are likely to be more subtle. Our regression analyses for predicting disability level at rehabilitation discharge provide support for the clinical validity of the WCST and the WCST-64. Although detecting and describing cognitive dysfunction is useful clinically, a test s capacity to predict patients functioning in the community can be particularly helpful to patients, families, and clinicians in treatment planning and disposition. Scores generated by each test were predictive of the outcome of interest. The predictive powers of the two tests were quite similar. There was only 2% difference in the amount of variability accounted for. These findings support the use of WCST-64 as a substitute for the WCST in early neuropsychological evaluations of patients with moderate and severe TBI. WCST-64 findings are substantially similar to WCST findings with regard to sensitivity to the neurocognitive impact of TBI and with regard to prediction of early outcome from TBI. Given the substantial time savings and decreased frustration for severely impaired patients who otherwise must complete all 128 trials of the standard WCST, any minimal loss of information occasioned by this substitution is warranted in early assessment of patients with significant injuries. An additional benefit of decreased exposure to WCST stimuli with the WCST-64 may be some mitigation of the practice effect for the WCST that has been so problematic for serial evaluations of all but the most severely impaired patients. Additional investigation of the WCST-64 is needed to further demonstrate its clinical utility for persons with TBI at later times postinjury and with other clinical populations. Investigation of the relationships of WCST-64 scores to other purported measures of executive functions such as the Controlled Oral Word Association Test, the Trail Making Test Part B, and the Category Test is also of interest.

8 WCST AND WCST ACKNOWLEDGMENT This research was supported by funding from the National Institute on Disability and Rehabilitation Research TBI Model Systems program grant numbers H133A980035, H133A011403, and H133A REFERENCES Anderson, S.W., Damasio, H., Jones, R.D., & Tranel, D. (1991). Wisconsin Card Sorting Test performance as a measure of frontal lobe damage. Journal of Clinical and Experimental Neuropsychology, 13, Axelrod, B.N., Henry, R.R., & Woodard, J.L. (1992). Analysis of an abbreviated form of the Wisconsin Card Sorting Test. The Clinical Neuropsycholgist, 6, Axelrod, B.N., Paolo, A.M., & Abraham, E. (1997). Do normative data from the full WCST extend to the abbreviated WCST? Assessment, 4, Boake, C., Millis, S.R., High, W.M., Jr., Delmonico, R.L., Kreutzer, J.S., Rosenthal, M., Sherer, M., & Ivanhoe, C.B. (2001). Using early neuropsychological testing to predict long-term productivity outcome from traumatic brain injury. Archives of Physical Medicine and Rehabililitation, 82, Clifton, G.L., Hayes, R.L., Levin, H.S., Michel, M.E., & Choi, S.C. (1992). Outcome measures for clinical trials involving traumatically brain-injured patients: Report of a conference. Neurosurgery, 31, Drewe, E.A. (1974). The effect of type and area of brain lesion on Wisconsin Card Sorting Test performance. Cortex, 10, Ferland, M.B., Ramsay, J., Engeland, C., & O Hara, P. (1998). Comparison of the performance of normal individuals and survivors of traumatic brain injury on repeat administrations of the Wisconsin Card Sorting Test. Journal of Clinical and Experimental Neuropsychology, 20, Gouvier, W.D., Blanton, P.D., LaPorte, K.K., & Nepomuceno, C. (1987). Reliability and validity of the Disability Rating Scale and the Levels of Cognitive Functioning Scale in monitoring recovery from severe head injury. Archives of Physical Medicine and Rehabilitation, 68, Graham, D.I. (1996). Neuropathology of head injury. In R.K.Narayan, J.E. Wilberger, & J.T. Povlishock (Eds.), Neurotrauma (pp ). New York: McGraw-Hill. Greve, K.W. (2001). The WCST-64: A standardized short-form of the Wisconsin Card Sorting Test. The Clinical Neuropsycholgist, 15, Hall, K.M., Cope, D.N., & Rappaport, M. (1985). Glasgow Outcome Scale and Disability Rating Scale: Comparative usefulness in following recovery in traumatic head injury. Archives of Physical Medicine and Rehabilitation, 66, Hannay, H.J., & Sherer, M. (1996). Assessment of outcome from head injury. In R.K. Narayan, J.E. Wilberger, & J.T. Povlishock (Eds.), Neurotrauma (1 ed., pp ). New York: McGraw-Hill. Heaton, R.K., Chelune, G.J., Talley, J.L., Kay, G.G., & Curtiss, G. (1993). Wisconsin Card Sorting Test Manual: Revised and Extended. Odessa, FL: Psychological Assessment Resources. Iverson, G.L., Slick, D.J., & Franzen, M.D. (2001). Clinical normative data for the WCST-64 following uncomplicated mild head injury. Applied Neuropsychology, 7, Kongs, S.K., Thompson, L.L., Iverson, G.L., & Heaton, R.K. (2000). WCST-64: Wisconsin Card Sorting Test-64 Card Version. Professional Manual. Odessa, FL: Psychological Assessment Resources. Kreutzer, J.S., Gordon, W.A., Rosenthal, M., & Marwitz, J.H. (1993). Neuropsychological characteristics of patients with brain injury: Preliminary findings from a multicenter investigation. Journal of Head Trauma Rehabilitation, 8, Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, Levin, H.S., O Donnell, V.M., & Grossman, R.G. (1979). The Galveston Orientation and Amnesia Test: A practical scale to assess cognition after head injury. Journal of Nervous and Mental Disease, 167, Little, A.J., Templer, D.I., Persel, C.S., & Ashley, M.J. (1996). Feasibility of the neuropsychological spectrum in prediction of outcome following head injury. Journal of Clinical Psycholology, 52, Milner, B. (1964). Some effects of frontal lobectomy in man. In J.M.Warren & K. Akert (Eds.), The frontal granular cortex and behavior (pp ). New York: McGraw-Hill. Najenson, T., Groswasser, Z., Mendelson, L., & Hackett, P. (1980). Rehabilitation outcome of brain damaged patients after severe head injury. International Rehabilitative Medicine, 2, Paolo, A.M., Axelrod, B.N., Troster, A.I., Blackwell, K.T., & Koller, W.C. (1996). Utility of a Wisconsin Card Sorting Test Short Form in persons with Alzheimer s and Parkinson s disease. Journal of Clinical and Experimental Neuropsychology, 18,

9 520 MARK SHERER ET AL. Rappaport, M., Hall, K.M., Hopkins, K., Belleza, T., & Cope, D.N. (1982). Disability Rating Scale for severe head trauma: Coma to community. Archives of Physical Medicine and Rehabilitation, 63, Rapport, L.J., Hanks, R.A., Millis, S.R., & Deshpande, S.A. (1998). Executive functioning and predictors of falls in the rehabilitation setting. Archives of Physical Medicine and Rehabilitation, 79, Robinson, A.L., Heaton, R.K., Lehman, R.A.W., & Stilson, D.W. (1980). The utility of the Wisconsin Card Sorting Test in detecting and localizing frontal lobe lesions. Journal of Consulting and Clinical Psychology, 48, Robinson, L.J., Kester, D.B., Saykin, A.J., Kaplan, E.F., & Gur, R.C. (1991). Comparison of two short forms of the Wisconsin Card Sorting Test. Archives of Clinical Neuropsychology, 6, Sherer, M., Madison, C.F., & Hannay, H.J. (2000). A review of outcome after moderate and severe closed head injury with an introduction to life care planning. Journal of Head Trauma Rehabilitation, 15, Smith-Seemiller, L., Franzen, M.D., & Bowers, D. (1997). Use of Wisconsin Card Sorting Test short forms in clinical samples. The Clinical Neuropsychologist, 11, Stuss, D.T. (1987). Contribution of frontal lobe injury to cognitive impairment after closed head injury: Methods of assessment and recent findings. In H.S. Levin, J. Grafman, & H.M. Eisenberg (Eds.), Neurobehavioral recovery from head injury (pp ). New York: Oxford University Press. Trudel, T.M., Tryon, W.W., & Purdum, C.M. (1998). Awareness of disability and long-term outcome after traumatic brain injury. Rehabilitation Psychology, 53,