The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery

Transcription

1 Schizophrenia Research 68 (2004) The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery Richard S.E. Keefe a, *, Terry E. Goldberg b, Philip D. Harvey c, James M. Gold d, Margaret P. Poe a, Leigh Coughenour a a Department of Psychiatry and Behavioral Sciences, Duke University Medical Center, P.O. Box 3270, Durham, NC 27710, USA b Clinical Brain Disorders Branch, National Institutes of Health, USA c Department of Psychiatry, Mount Sinai School of Medicine, USA d Maryland Psychiatric Research Center, USA Received 21 February 2003; received in revised form 9 September 2003; accepted 10 September 2003 Abstract Studies of neurocognitive function in patients with schizophrenia use widely variable assessment techniques. Clinical trials assessing the cognitive enhancing effect of new medications have used neurocognitive assessment batteries that differed in content, length and administration procedures. The Brief Assessment of Cognition in Schizophrenia (BACS) is a newly developed instrument that assesses the aspects of cognition found to be most impaired and most strongly correlated with outcome in patients with schizophrenia. The BACS requires less than 35 min to complete in patients with schizophrenia, yields a high completion rate in these patients, and has high reliability. The BACS was found to be as sensitive to cognitive impairment in patients with schizophrenia as a standard battery of tests that required over 2 h to administer. Compared to healthy controls matched for age and parental education, patients with schizophrenia performed 1.49 standard deviations lower on a composite score calculated from the BACS and 1.61 standard deviations lower on a composite score calculated from the standard battery. The BACS composite scores were highly correlated with the standard battery composite scores in patients (r = 0.76) and healthy controls (r = 0.90). These psychometric properties make the BACS a promising tool for assessing cognition repeatedly in patients with schizophrenia, especially in clinical trials of cognitive enhancement. D 2003 Elsevier B.V. All rights reserved. Keywords: Schizophrenia; BACS; Neurocognitive assessment batteries 1. Introduction The cognitive deficits of schizophrenia are profound and devastating. Patients with chronic schizophrenia demonstrate impairments that range between * Corresponding author. Fax: address: Richard.keefe@duke.edu (R.S.E. Keefe). one and a half to two standard deviations below healthy controls on several key dimensions of cognition, especially verbal memory, working memory, motor speed, attention, executive functions and verbal fluency (Saykin et al., 1991; Keefe, 1995; Harvey and Keefe, 1997; Heinrichs and Zakzanis, 1998). These components of cognitive dysfunction in schizophrenia also have an impact on functional outcome (Green, /$ - see front matter D 2003 Elsevier B.V. All rights reserved. doi: /j.schres

2 284 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) ; Green et al., 2000) as they are all correlated with poor functional abilities. Since the second generation of antipsychotic medications became widely available, dozens of studies of the neurocognitive impact of these medications have been completed. While most of these studies have concluded that second generation antipsychotics improve neurocognitive function, the interpretation of these results has been challenged, in part, by the variable test batteries used in each study (Harvey and Keefe, 2001). There is no standard, easily administered test battery that specifically and efficiently assesses the important cognitive deficits in patients with schizophrenia. Instead, current test batteries differ widely in content, duration, procedures, and implementation. The limited generalizability of neurocognitive findings across studies has reduced the ability of clinicians and researchers to make clear conclusions about the relative impact of the various antipsychotic medications on cognition in schizophrenia. The absence of a standard measure has also challenged studies of adjunctive treatments for cognition in patients with schizophrenia (Friedman et al., 2001, 2002; Evins et al., 2000; Heresco- Levy et al., 1996). Another drawback of current assessments of cognition in treatment studies of patients with schizophrenia is that many of the neurocognitive assessment batteries used are long and complex. Most neuropsychological assessment batteries used in schizophrenia studies have been adapted from clinical neuropsychology, which assesses the entire profile of neuropsychological strengths and weaknesses in individuals. These batteries of tests may require several hours to administer. Their adaptation for schizophrenia research has kept some of the length that was originally necessary for individual assessment, but may no longer be necessary for research such as clinical trials that compares groups of patients. In fact, the length of some assessment batteries may be a limiting factor in assessing patients repeatedly throughout a clinical trial (Harvey and Keefe, 2001). In contrast to these standard assessment techniques in schizophrenia research and clinical practice, cognitive function in patients with dementia is usually assessed with a widely used cognitive assessment tool, such as the Mini Mental Status Examination (Folstein et al., 1975), Dementia Rating Scale (Gardner et al., 1981), or Alzheimer s Disease Assessment Scale (Rosen et al., 1984). These instruments can be easily administered in typical treatment settings such as a physician s office, and are routinely employed as indicators of treatment change in clinical trials of anti-dementia drugs. The administration of These tests shed light on the global severity of patients cognitive deficits, track progression, and measure cognitive changes. The availability of a quick and efficient tool for measuring cognition in patients with schizophrenia could be an extremely useful guide for clinicians making decisions about potential rehabilitation and for researchers implementing clinical trials to assess cognitive improvement. There are several batteries of tests that are available or in development for the purposes of brief cognitive assessment. Several computerized batteries, such as the Cambridge Neuropsychological Test Automated Battery (CANTAB) (Robbins et al., 1996), the CDR Cognitive Assessment System (Hunter et al., 1997), and the CogTest Battery (Cogtest, 2002) have been applied to schizophrenia samples, and have the option of reduced length. However, portability, regular software and hardware version changes, and increased patient and tester burden present implementation challenges. Other batteries, such as the Cognitive Screening Instrument for Schizophrenia (CSIS), a very short (10 min) set of abbreviated tests (Scot Purdon, personal communication), are currently in development. The work completed with the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (Randolph, 1998; Gold et al., 1999; Hobart et al., 1999) has clearly demonstrated the utility of brief assessment approaches. The RBANS is capable of providing reliable and valid assessments of patients with schizophrenia in various cognitive domains (Gold et al., 1999; Hobart et al., 1999; Wilk et al., 2002). As noted in the RBANS manual, however, the test was originally developed as a screening measure primarily for elderly subjects, and the test is heavily weighted by memory, language, and visual perceptual subtests. In addition, the item difficulties are most relevant to the types of impairment likely to be observed in patients with dementing illnesses such as Alzheimer s Disease, with ceiling effects in some domains. Although the

3 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) RBANS is clearly sensitive to some of the impairments observed in schizophrenia, and it has utility as a screening tool for patients with schizophrenia, it lacks measures of motor, executive, and working memory performance that may be particularly important targets for cognitive enhancement in schizophrenia. These limitations suggest the need for a measure specifically designed for use in schizophrenia clinical trials that preserves the desirable features of the RBANS: brief administration and scoring time, portablity, repeatability, and availability of alternate forms. The Brief Assessment of Cognition in Schizophrenia (BACS) has been developed for clinical trials with these key features. In addition, the domains of cognitive function that are assessed by the BACS are those found to be consistently impaired, and consistently related to outcome, in schizophrenia: verbal memory, working memory, motor speed, attention, executive functions and verbal fluency. The BACS is fully portable, and is designed to be easily administered by a variety of testers, including nurse clinicians, psychiatrists, neurologists, social workers, and other mental health workers. It is designed to require about 30 min of testing time with minimal extra time for scoring and minimal training demands. The development of a psychological assessment instrument should determine the instrument s test retest reliability, sensitivity, criterion-referenced validity (i.e. comparison to a standard measure), and comparability of alternative forms. This study aims to determine the quality of the BACS in these domains of reliability and validity. 2. Methods 2.1. Subjects Healthy controls were recruited from a variety of sources, including bulletin board postings, newspaper advertisements, and word of mouth. They were interviewed with the selected sections of the Structured Clinical Interview for DSM-IV Axis I Disorders, Clinical Version, for healthy people and were required not to have an Axis I disorder according to DSM-IV criteria, nor any relevant neurologic illness such as a history of brain trauma. Patients were recruited from the inpatient and outpatient facilities at Duke University, John Umstead Hospital, the University of North Carolina Neurosciences Hospital, and Dorothea Dix Hospital. Patients were required to meet DSM-IV criteria for schizophrenia, schizoaffective illness, or schizophreniform disorder, to have no history of brain trauma, nor to be suffering from a current substance use disorder. There were no specific medication criteria for inclusion in the patient group. Ninety-four of the 150 patients were being treated with a single atypical antipsychotic medication (31 with risperidone, 28 with olanzapine, 15 with clozapine, 10 with aripiprazole, 8 with quetiapine, and 2 with ziprasidone), 13 were being treated with a single typical antipsychotic (6 with haloperidol, 4 with haldol decanoate, 1 with prolixin, 1 with loxitane, 1 with navane), 9 with a combination of antipsychotics, and 8 with an antipsychotic that was not currently known because the patient was in a double-blind study. Ten were receiving pharmacologic treatment but not with antipsychotics (e.g. ativan, lithium), and for 16 patients, medication information was not available. The demographic characteristics of the patients with schizophrenia and the healthy controls are described in Table 1. Study investigators made a concerted effort to recruit healthy controls who would match the patients on parental education, age, ethnic background, and sex. As indicated in Table 1, groups did not differ significantly on any of these measures except sex. Males were more highly represented in the patient group (79%) than in the control group (62%)(Fisher s exact, P = 0.023) Assessment procedures Subjects were tested on two separate days, with no more than 3 days between assessments. Subjects were randomly assigned to a sequence of BACS versions A and B. On the first test session, subjects received one version of the BACS followed by the first half of the standard battery, which included the following tests: Rey Auditory Verbal Learning Test (Geffen et al., 1994), Dot Test of Visuospatial Working Memory (Keefe et al., 1997), Grooved Pegboard (Ruff and Parker, 1993), Wide Range Achievement Test, third edition, Reading subtest (Wilkinson, 1993), Trail-

4 286 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) making A (Reitan, 1979), Trailmaking B (Reitan, 1979) and Letter Number Test of Auditory Working Memory (Gold et al., 1997). On the second test session, subjects were randomized to receive either the same or different version of the BACS followed by the second half of the standard battery, which included these tests: Wechsler Memory Scale, third edition, Logical Memory subtest (Wechsler, 1997b), Controlled Oral Word Association Test (Lezak, 1995), Category Instances (Lezak, 1995), Wechsler Adult Intelligence Scale, third edition, subtests (Digit Symbol-Coding, Block Design, Arithmetic, and Information) (Wechsler, 1997a), Continuous Performance Tests (A-X, two-digit Identical Pairs and four-digit Identical Pairs) (Cornblatt and Keilp, 1994), and computerized Wisconsin Card Sort Test (64 card version) (Heaton, 1993). The constructs measured with the BACS, including the tests, procedures, and measures are listed in the order administered as follows Verbal memory List learning. Patients were presented with 15 words and then asked to recall as many as possible. This procedure was repeated five times. There were two alternate forms. Measure: number of words recalled per trial, in any order (range: 0 75) Working memory Digit sequencing task. Patients were presented with clusters of numbers of increasing length. They were asked to tell the experimenter the numbers in order, from lowest to highest. Measures: number of correct responses (range: 0 28); longest sequence recalled correctly (range: 0 8) Motor speed Token motor task. Patients were given 100 plastic tokens and asked to place them two at a time into a container as quickly as possible. A 60-s time limit was imposed. Measures: the number of tokens correctly placed into the container for the first halfminute, second half-minute, and the 1 min total (range: 0 100) Verbal fluency Category instances. Patients were given 60 s to name as many words as possible within a given category. Version A: supermarket items; Version B: tools Controlled oral word association test. In two separate trials, patients were given 60 s to Table 1 Demographics of the sample Measure Schizophrenia Controls t P N Mean S.D. N Mean S.D. Age Education (years) Paternal education Maternal education WRAT-3, Reading Sex N (%) a Male 119 (79) 31 (62) Female 31 (21) 19 (38) Race N (%) a African 70 (50) 28 (56) Caucasian 65 (46) 22 (44) Other 5 (4) 0 (0) WRAT-3 = Wide Range Achievement Test, Third edition. a By Fisher s exact test.

5 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) generate as many words as possible that begin with a given letter. Version A: F, S; Version B: P, R Measure: number of words generated per trial Attention and speed of information processing Symbol coding. As quickly as possible, patients wrote numerals 1 9 as matches to symbols on a response sheet for 90 s. Measure: number of correct numerals (range: 0 110) Executive functions Tower of London. Patients were shown two pictures simultaneously. Each picture showed three balls of different colors arranged on three pegs, with the balls in a unique arrangement in each picture. Patients were asked to give the total number of times the balls in one picture need to be moved in order to make the arrangement of balls identical to that of the other, opposing picture. There were 20 trials. The items were of variable difficulty, with a general tendency for later items to be more difficult. The test was discontinued if patients made five consecutive incorrect responses. If patients responded correctly to all 20 trials, two additional trials of greater difficulty were administered. There were two alternate forms. Measure: number of correct responses (range: 0 22) Data analyses For all subtests and composite scores, test retest reliability was measured with intra-class correlations (ICC) in the patient and control groups separately. The ICC is a conservative estimate of test retest reliability, as it is sensitive to group mean changes over time in addition to intra-subject variability. Practice effects were measured by comparing data collected at test session 1 to those collected at test session 2 with within-group t-tests. These were determined in the patient and control groups separately. Sensitivity to between-group impairment on all measures was determined with independent t-tests. The primary measure from each test of the BACS was standardized by creating z-scores whereby the test session 1 healthy control mean was set to zero and the standard deviation set to one. Tests with alternate forms were standardized separately for each version. A composite score was calculated by averaging all of the six standardized primary measures from the BACS, and then calculating a z-score of the composite. Composite scores were calculated for all subjects using the a priori criterion that they must have successfully completed all or all but one of the measures that comprised the composite score. This criterion included the entire sample, as no subjects were missing more than one BACS measure. The relationship among the BACS measures was determined by calculating Pearson correlations among the scores. The factor structure of the scores was determined by performing a principal components analysis with oblique rotation. For the standard battery, a composite score was calculated by adding together the primary measures from each test after they had been standardized. Standardized scores for each construct (e.g. verbal memory) were determined by calculating the mean of the standardized scores for each of the measures that comprised the construct. A z-score for each measure was calculated based upon the healthy control mean and standard deviation, and the average of the z-scores was calculated to determine the construct score. The Dot Test working memory deficit score, Trailmaking A and B scores, and WCST perseverative error scores were multiplied by 1 so that good performance was associated with scores in a positive direction. The following constructs were calculated using the measures listed in parentheses: Verbal memory (RVLT total scores, WMS-III logical memory score), Attention (WAIS-III digit symbol, Mean CPT d-prime), Working Memory (Dot test working memory error, number of correct items on Letter Number Sequencing), Motor Speed (Mean of dominant and nondominant hand Pegboard performance, Trailmaking A), Verbal Fluency (COWAT, Category instances), Executive functions (Mean of WCST categories and perseverative error z-scores, Trailmaking B). The relative sensitivity of the composite scores derived from the standard battery and the BACS was determined by comparing each of their betweengroup differences with t-tests. In addition, multiple logistic regression analyses were performed to determine the amount of unique between-group variance that could be accounted for by each battery.

6 288 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) Finally, to compare the relation between the BACS measures and the measures from the standard battery, the neurocognitive construct summary scores from the standard battery were correlated with the BACS measures using Pearson correlations. 3. Results 3.1. Description of BACS data Test retest reliability, sensitivity and practice effects of tests without alternate forms Table 2 lists the means and standard deviations for all of the measures from the BACS that do not have alternate forms. These measures were repeated with the same form regardless of which version of the BACS was administered. All tests demonstrated significant differences ( P < 0.001) between controls and patients. The intra-class correlations (ICC) between performance from test session 1 and test session 2 for each measure are also included. Each test had one measure that produced an ICC of 0.79 or greater for the patient group and healthy control group: total number of correct responses from the digit sequencing task; correct responses from the symbol coding task; and total number of tokens for 60 s from the token motor task. The longest correct sequence measure for the digit sequencing tests and the 30-s measures for the token motor task produced ICCs that were inferior to the primary measures, and were thus not used in subsequent analyses. The effect of practice on the primary measures was small. Only the symbol coding test showed significant practice effects in patients (0.25 S.D.) and controls (0.34 S.D.) Comparability of versions: test retest reliability, sensitivity, and practice effects of tests with alternate forms Table 3 lists the means and standard deviations for the BACS measures derived from tests with alternate forms. The data are organized by the sequence of the BACS versions subjects received in test sessions 1 and 2. The ICCs between performance for test session 1 and test session 2 for each measure are also included. Subjects who received the same version twice (AA and BB) allow for an assessment of maximal potential practice effects and test retest reliability. Subjects who received a different version on consecutive test sessions (AB and BA) allow an assessment of general practice effects that are not dependent upon test version, as well as the comparability of the test forms Verbal memory. The word lists from each version were very similar in difficulty in patients and controls. The final versions of the lists were the result of a reorganization of the original word lists accomplished by switching six of the words between the A list and the B list, keeping the words in the same or similar ordinal position as the original list. Since the samples were smaller for the comparison of the final lists, we collected data on an additional 30 controls from Mount Sinai Medical Center not included in the Table 2 Mean performance and reliability coefficients of BACS tests with no alternate forms in patients with schizophrenia and healthy controls Measure Schizophrenia Controls P1 P2 Test session 1 Test session 2 ICC Test session 1 Test session 2 ICC N Mean S.D. N Mean S.D. N Mean S.D. N Mean S.D. Digit sequencing, total correct Digit sequencing, longest correct sequence Token motor task (1st 30 s) Token motor task (2nd 30 s) Token motor task total Symbol coding P1 = Significance value for patients day 1 vs. patients day 2, by t-test; P2 = Significance value for controls day 1 vs. controls day 2, by t-test. Significance value for all patients day 1 vs. controls day 1 comparisons, P < 0.001, by t-test; Significance value for all patients day 2 vs. controls day 2 comparisons, P < 0.001, by t-test.

7 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) original sample. In this separate sample, the word list difficulty was almost identical, and not significantly different (A list: mean = 50.33, S.D. = 9.77; B list mean = 51.00, S.D. = 9.48; t = 0.19, df = 28, n.s.). The ICCs for this measure ranged between 0.78 and 0.93 in patients and 0.40 and 0.90 in controls. Due to a randomization procedure that resulted in fewer than expected controls with an AB or BA sequence of tests using the final word list, we administered the word lists only to an additional 11 controls not otherwise included in the study to increase the sample size for assessing the test retest reliability in controls receiving different versions. The practice effect was significant in patients and controls who were administered the same version, but not in subjects who received different versions on consecutive test sessions. Controls performed significantly better than patients for each version of the test. These results support the need for alternate forms for this test Verbal fluency. Letter Fluency: In subjects receiving the same version on consecutive test sessions, the ICCs were good for the patient group, ranging between 0.69 and 0.78, and lower for the controls, ranging between 0.53 and In subjects Table 3 Mean performance and reliability coefficients of BACS tests with alternate forms in patients with schizophrenia and healthy controls Condition Schizophrenia Controls Significance Test session 1 Test session 2 ICC Test session 1 Test session 2 ICC P1 P2 P3 P4 N Mean S.D. N Mean S.D. N Mean S.D. N Mean S.D. Verbal memory A/A B/B A/B B/A Verbal fluency A/A F S Supermarkt B/B P R Tools A/B F/P S/R Sup/Tools B/A P/F R/S Tools/Sup Tower of London A/A B/B A/B B/A P1 = Significance value for patients test session 1 vs. patients test session 2, by t-test; P2 = significance value for controls test session 1 vs. controls test session 2, by t-test; P3 = significance value for patients test session 1 vs. controls test session 1, by t-test; P4 = significance value for patients test session 2 vs. controls test session 2, by t-test.

8 290 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) Table 4 Mean performance and reliability coefficients of BACS composite scores in patients with schizophrenia and healthy controls Condition Schizophrenia Controls Significance Test session 1 ICC Test session 2 Test session 1 ICC Test session 2 P1 P2 P3 P4 N Mean S.D. N Mean S.D. N Mean S.D. N Mean S.D. A/A B/B A/B B/A P1 = Significance value for patients test session 1 vs. patients test session 2, by t-test; P2 = significance value for controls test session 1 vs. controls test session 2, by t-test; P3 = significance value for patients test session 1 vs. controls test session 1, by t-test; P4 = significance value for patients test session 2 vs. controls test session 2, by t-test. receiving different versions, the ICCs were variable, but poor overall, with three ICCs below 0.17 and none higher than The practice effect was small in those receiving the same version on consecutive test sessions (less than 0.33 S.D. for all letters), and was not statistically significant in controls and patients for F and S words. Performance for patients was significantly different from controls for each of the letters. Category Instances: The ICCs for subjects receiving the same version on consecutive test sessions were good, ranging from 0.71 to 0.90, but poor in subjects receiving different versions on consecutive test sessions ( in patients; in controls). Patients and controls performed about 2 S.D. better on Version A (supermarket items) than Version B (tools). The practice effect was statistically significant for the tools category, but not for supermarket items in controls. The pattern of between-group results suggested that the category of supermarket items was more sensitive to the differences between patients and controls than tools. These results suggest that the category of supermarket items is a superior measure Tower of London. Versions A and B were very similar in patients and controls, with no significant differences between versions. The ICCs ranged between 0.66 and 0.89 in patients and between 0.73 and 0.96 in controls. The practice effect was small in patients ( < 0.25 S.D.) and medium in controls ( S.D.). The sensitivity of the test to the differences between patients and controls was weaker than the other measures, although they were statistically significant in the larger AA and BB samples BACS composite scores and BACS profile Table 4 lists the composite scores for controls and patients receiving the various combinations of the two versions of the BACS. Note that all data were standardized using test session 1 means and standard Fig. 1. Performance of patients with schizophrenia on the BACS subtests and composite score standardized to healthy controls. All differences between patients and controls were statistically significant ( P < 0.001).

9 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) Table 5 Correlations among BACS measures for healthy controls and schizophrenia patients VM DS VF SC TM TL Comp Composite score Tower of London Token motor Symbol coding Verbal fluency Digit sequencing Verbal memory Correlations in controls above diagonal; correlations in patients below diagonal. Among patients, correlations greater than or equal to 0.29, P < 0.001; , P < 0.01; , P < Among controls, correlations greater than or equal to 0.45, P < 0.001; , P < 0.01; , P < VM = Verbal memory; DS = Digit sequencing; VF = Verbal fluency; SC = Symbol coding; TM = Token motor task; TL = Tower of London; Comp = Composite score. deviations, therefore a slight practice-related improvement for test session 2 data was expected. Test retest reliability was very high, even in subjects receiving different versions between test sessions. The ICCs ranged between 0.86 and 0.92 for patients, and between 0.87 and 0.95 for healthy controls. The differences between patients and controls were statistically significant for all versions and groups with the exception of those who received the tests in the B/A order. The between-group difference of the composite score for this group ( P = 0.054) appears to have been smaller than the others due to a combination of factors, including poor overall performance in this group of eight controls and better overall performance in this patient group. Fig. 1 describes the performance of patients on each of the primary measures of the BACS and the Table 6 Factor loadings of BACS measures in patients with schizophrenia Component Digit sequencing Symbol coding Tower of London Token motor task Verbal fluency Verbal memory Eigenvalue Percentage of variance composite score compared to the healthy control mean. In this figure, the data from versions A and B were collapsed together to include all subjects. All differences between patients and controls were statistically significant ( P < 0.001). The BACS composite scores did not differ significantly between males (mean = 1.52 F 1.09) and females ( 1.37 F 1.10) with schizophrenia. Differ- Table 7 Performance of patients with schizophrenia and healthy controls on standard battery of tests Schizophrenia Controls N Mean S.D. Z score N Mean S.D. RAVLT total Dot average Pegboard dominant Pegboard non-dominant Mean pegboard Trail making A Trail making B Letter number sequencing total Letter number sequencing longest Logical memory mean A and B COWAT mean Category fluency mean WAIS-III information WAIS-III blocks WAIS-III arithmetic WAIS-III digit symbol CPT A-X CPT 2 digits CPT 4 digits CPT mean WCST perseverative errors WCST-categories RAVLT = Rey Auditory Verbal Learning Test, COWAT = Controlled Oral Word Association Test, WAIS-III = Wechsler Adult Intelligence Test, third edition, CPT = Continuous Performance Test, WCST = Wisconsin Card Sorting Test.

10 292 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) Fig. 2. Composite scores for the BACS and standard battery in schizophrenia patients standardized to healthy controls. ences between males (0.27 F 1.05) and females ( 0.45 F 0.71) (t = 2.65, df =48, P = 0.011) in the healthy control group were consistent with the significantly higher education and WRAT-Reading scores for the males in this sample Correlations among BACS measures and factor analysis Table 5 presents the correlations among the primary BACS measures for patients (below the diagonal) and controls (above the diagonal). All but 3 of the 42 correlations were significant at the P < 0.05 level. The correlations were of slightly greater magnitude in the controls, yet the pattern of correlations between the groups were very similar. The pattern of correlations between each of the measures and the composite scores, although greater in the controls, was strikingly similar between groups. Principal components analysis with oblique rotation was completed to determine the factor structure of the BACS. The factor loadings are presented in Table 6. The factor structure suggests a three-factor solution. Measures that emphasize motor speed and general cognitive functions load on the first factor; the memory and working memory measures load on the second factor, and executive function loads on the third factor Description of standard battery data Table 7 presents the performance of schizophrenic patients and controls on the standard battery of tests. The table includes the group mean and standard deviation for the primary measure of each test and the z-scores for the patients. The least sensitive measures in the battery were letter number sequencing and WCST categories, coinciding with the two least sensitive measures in the BACS, digit sequencing (measuring verbal working memory) and the Tower of London test (measuring executive function) Relationship of standard battery and BACS Testing duration The BACS required a mean of 34.2 min for patients (S.D. = 8.95) and 30.4 (S.D. = 3.88) min for Table 8 Pearson correlations for standard battery domains and BACS measures in healthy controls Long battery BACS measures domains VM DS VF SC TM TL Comp Verbal memory Attention Working memory Motor speed Verbal fluency Executive function Composite score BACS measures: VM = Verbal memory; DS = Digit sequencing; VF = Verbal fluency; SC = Symbol coding; TM = Token motor task; TL = Tower of London; Comp = Composite score. Correlations greater than and or equal to 0.48, P < 0.001; , P < 0.01; , P < 0.05.

11 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) Table 9 Pearson correlations for standard battery domains and BACS measures in schizophrenia patients Long battery BACS measures domains VM DS VF SC TM TL Comp Verbal memory Attention Working memory Motor speed Verbal fluency Executive function Composite score BACS measures: VM = Verbal memory; DS = Digit sequencing; VF = Verbal fluency; SC = Symbol coding; TM = Token motor task; TL = Tower of London; Comp = Composite score. Correlations greater than or equal to 0.29, P < 0.001; , P < controls on the first day of testing. Although data were not collected in a standard manner for all subjects, time estimates for a small sample of subjects (N = 10 for each group) suggested that the standard battery required an average of 133 min (S.D. = 43.1) for patients and 109 min (S.D. = 20.0) for controls Completion rate A total of 140 of the 150 patients completed each of the subtests of the BACS and the pencil- and -paper tests of the standard battery. Only 115 patients completed the CPT and 117 patients completed the WCST, primarily due to patient refusal or inability to follow test instructions; both of these tests were computerized Sensitivity to deficits Fig. 2 demonstrates the relative sensitivity of the composite scores from the standard battery and the BACS during test session 1. BACS data from test session 2 were not included to minimize any effects of practice. The difference between the composite scores from the BACS and the standard battery was not significantly different. Multivariate logistic regression analyses were performed to determine the sensitivity of the standard battery beyond what could be provided by the BACS. BACS composite scores accounted for 27.1% of the between-group variance in diagnosis ( F = 73.61; df = 1,198; P < 0.001). The standard battery accounted for 3.3% additional variance, which was a small but statistically significant variance component ( F = 9.46; df = 1, 197; P < 0.01). However, when the standard battery composite score was entered as the first step in the equation, and the BACS composite scores were entered second, the BACS composite scores also accounted for a small but statistically significant 1.5% of additional variance ( F =4.31; df =1, 197; P < 0.05). Fig. 3. Scatterplot of BACS and standard battery composite scores for patients and controls.

12 294 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) When only those subjects who received version A at test session 1 were included in the analysis, BACS composite scores accounted for 32.6% of the betweengroup variance in diagnosis ( F = 37.38; df = 1,98; P < 0.001). The standard battery composite score accounted for only 0.1% additional variance, which was not statistically significant. When the order of entry was reversed, BACS version A composite scores accounted for 5.3% additional variance ( F = 36.95; df = 1,98; P < 0.01) beyond the standard battery composite scores. When only subjects who received version B at test session 1 were included in the analysis, BACS composite scores accounted for 23.0% of the betweengroup variance in diagnosis ( F = 29.22; df = 1,98; P < 0.001). The standard battery composite score accounted for 7.6% additional variance ( F = 10.65; df = 1,97; P < 0.01). When the order of entry was reversed, BACS version B composite scores accounted for only 0.1% additional variance, which was not statistically significant Correlations between BACS measures and standard battery constructs The correlations between the standard battery constructs and the BACS measures (Test session 1 data only) are presented in Table 8 for controls and Table 9 for patients. The correlations were slightly higher in the controls, yet the pattern of correlation magnitude was similar between groups. The highest correlation in each matrix was between the standard battery and BACS composite scores, which was 0.76 for patients and 0.90 for controls. The individual data points from this correlation are presented in Fig Discussion The Brief Assessment of Cognition in Schizophrenia, or BACS, an easily administered pen- and -paper battery of neurocognitive tests, demonstrated high reliability and concurrent validity with a standard battery of tests in schizophrenia patients and healthy controls with similar ages, racial backgrounds and parental education. The BACS was as sensitive to the cognitive deficits of schizophrenia as a standard battery of neuropsychological tests that took nearly four times as long to administer. Further, a greater percentage of patients were able to complete the tests in the BACS compared to the standard measures, especially those administered with computerized methods. The BACS offers promise as a tool for assessing cognition repeatedly over the course of treatment in patients with schizophrenia Psychometrics The intraclass correlation coefficients calculated from the data in this study suggested that the composite scores derived from each subtest had very good test retest reliability in patients with schizophrenia and healthy controls. These reliability coefficients were high not only when the same version of the test was administered on consecutive test sessions, but also when different versions (alternate forms) of the test were administered. The benefit of high reliability on the BACS composite score is the increased likelihood that change over time on this measure, such as with treatment, can be interpreted as due to a nonrandom effect. The primary measures from the subtests without alternate versions, including tests of motor speed, working memory, verbal fluency and information processing speed, were highly reliable, with ICCs equal to or greater than The practice effects of these tests were minimal, with none of the improvements exceeding 0.25 standard deviations, despite the potential effect of practice being maximized by testing twice within 5 days. The comparisons of the subtests from the alternate forms suggested that alternate forms for the verbal memory and Tower of London tests are necessary, but a single version measuring verbal fluency is sufficient. Regarding verbal memory, the lists proved to be very similar in difficulty and highly reliable. The practice effect across these versions was very small compared to the practice effect using the same versions. Thus, the use of alternate forms for the verbal memory test is necessary, and will facilitate the assessment of changes in verbal memory abilities that are independent from learning the words in a previous administration. The reliability of the verbal fluency measures was found to be much higher when the same versions were used on consecutive test sessions. The effect of practice on these measures was very small, even when the same version was administered on consec-

13 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) utive test sessions. In addition, there was variable sensitivity of the different versions, particularly in the domain of category fluency, with supermarket items being the most sensitive. This pattern of results suggests that with regard to category instances, supermarket items will be the most reliable and sensitive category to use for both versions of the BACS. There was little differentiation in reliability, practice effects, and sensitivity to group differences among the various letter categories for the Controlled Oral Word Association Test. It appears as though any combination of letters is satisfactory as long as it is consistent on consecutive testing periods. The reliability of the Tower of London was high, even when different versions were administered on consecutive test sessions. The practice effects were small in patients and medium in controls when the same versions were used on consecutive test sessions. However, these practice effects were diminished when a different version was used on consecutive test sessions. These data suggest that using an alternate form for this test is helpful to reduce practice effects, and recommended. The pattern of results regarding the reliability, practice effects, and between-group sensitivity of tests that had alternate forms in this study suggest that the final BACS should include alternate forms for verbal memory and Tower of London tests only. The tests without alternate forms digit sequencing, symbol coding, and the token-motor task had minimal practice effects. Due to the reduced reliability that results from alternate forms of verbal fluency measures, and due to the minimal practice effects that result from administering the same version consecutively, the verbal fluency tests should not have alternate forms. Based upon the slightly higher reliability, smaller practice effect, and increased sensitivity to betweengroup differences, the best verbal fluency categories to use in the BACS are F and S for letter fluency and supermarket items for category fluency. Thus, the final version of the BACS that is recommended is one in which alternate forms are used for verbal memory and Tower of London subtests only, and a single version of verbal fluency is used that includes supermarket items, F-words and S-words. It should also be noted that all of the final measures of the BACS have distributional properties suggesting minimal ceiling- and floor-effects. These properties are important for assessing change over time, such as in clinical trials Validity The composite scores from the BACS and the standard battery were highly correlated in patients and in controls, and neither battery was more sensitive to the overall deficits found in patients with schizophrenia. The magnitude of these deficits, which was about 1.5 standard deviations below the healthy controls in this study, were consistent with those reported in meta-analyses of studies of neurocognitive impairment in schizophrenia (Heinrichs and Zakzanis, 1998), yet not as severe as estimated from a more selective review of the literature (Harvey and Keefe, 1997). Studies such as this one that pay special attention to matching groups on age and parental education may yield differences that are less robust than those that do not attend to these factors (Heinrichs and Zakzanis, 1998). The differences between patients and controls may appear to be smaller than expected on the Tower of London test, measuring executive functions, and the digit sequencing test, which measures a verbal form of working memory. While these measures may initially appear to be less sensitive to the neurocognitive deficits of schizophrenia, it is possible that the between-group differences on these measures were relatively small due to the particular cohorts tested, as the measures from the standard battery used to assess these cognitive domains were similarly small in their differences between patients and controls. Furthermore, the magnitude of these deficits are consistent with the effect sizes reported in the meta-analysis by Heinrichs and Zakzanis (1998). The factor analyses and correlations among the BACS measures suggests that while there is a single general factor of cognitive performance that can be derived from the BACS raw scores, there are two other relatively dissociable domains of performance measured by the BACS. The pattern of correlations among the BACS measures was very similar in patients and controls. In both groups, each of the individual measures demonstrated high correlations with the composite scores. Factor analysis conducted on the data collected only from patients suggests that the first factor, which accounted for the largest amount of variance, is a factor of general cognitive

14 296 R.S.E. Keefe et al. / Schizophrenia Research 68 (2004) performance with an emphasis on speed and the generation of action. The other two factors appeared to reflect more discrete functions, with the second factor including measures of memory and the third including executive functions. The BACS thus enables an assessment of overall cognitive function as well as scores on individual cognitive domains. The definitive validation of the BACS will require further study. First, the comparisons of the final word lists for the verbal memory test had sample sizes that were reduced, which resulted in less statistical power available for these analyses. However, the final lists appear to have remarkable similarity in their sensitivity to group differences. This similarity was also found in a separate group of controls that was not a part of this study sample. Second, we chose to test subjects twice within 1 week to minimize the impact of changes in clinical state and medication, reduce drop out, and maximize practice effects. However, the assessment of a treatment effect in research studies or clinical purposes is likely to be measured with longer time between assessments. Thus, the reliability of these measures in actual practice may be lower and the practice effects of the tests in the BACS are likely to be smaller. Third, one of the drawbacks of a focus on composite scores in the evaluation of cognition in schizophrenia is that isolated yet important cognitive effects may be missed. It is possible that a new medication may improve an important aspect of cognition that is not assessed by the BACS. However, the importance of general cognitive effects can be seen in the relative size of the correlations between functional outcome and different aspects of cognition. The Pearson correlations between outcome and individual aspects of cognition such as memory, attention, and executive functions are relatively small, ranging between 0.20 and 0.34 (Green, 2002) The few studies that have enabled the determination of the correlation of outcome with overall summary scores of cognition have demonstrated far larger correlations, exceeding 0.60 (Green, 2002). These data suggest that the determination of cognition with summary data, such as the BACS composite scores reported here, may be the most powerful indicator of functional outcome, and has the most promise as a target for cognitive enhancement that will produce real-world changes in patients lives outside the laboratory. Finally, a complete validation of the BACS will require assessments in various populations of patients with schizophrenia, such as first episode populations, treatment refractory schizophrenia patients, and geriatric patients. It will also be important to determine whether BACS scores can predict changes in functional outcome, such as activities of daily life, and whether the BACS is sensitive to cognitive changes during clinical trials. Studies are underway to determine the validity of the BACS in these areas of inquiry. In sum, the BACS assesses the major constructs of cognition that have been found to be most impaired and most strongly correlated with outcome in patients with schizophrenia. The BACS takes less than 35 min to complete in patients with schizophrenia, yields a high completion rate in these patients, and has high test retest reliability over a period of days. It is as sensitive to cognitive impairment in patients with schizophrenia as a standard battery of tests that required over 2 h to complete. While the sensitivity of the BACS to change over time can only be determined through longitudinal treatment studies, 10 of which are currently underway, its psychometric properties make it a promising tool for assessing cognition repeatedly in patients with schizophrenia. Acknowledgements This work was supported by a grant from Eli Lilly. Mark Appelbaum provided statistical consultation. Neurocognitive data were collected, in part, by Adam Vaughn, Susan Shortel, Matthew Dukes, Trina Walker and Joseph Kang. Healthy controls at Mount Sinai were assessed by Adam Brickman and Julie Kim. References Cogtest plc. Cogtest(tm): Computerised Cognitive Battery for Clinical Trials. (2002). Retrieved November 8, 2002, from Cornblatt, B.A., Keilp, J.G., Impaired attention, genetics, and the pathophysiology of schizophrenia. Schizophr. Bull. 20, Evins, A.E., Fitzgerald, S.M., Wine, L., Rosselli, R., Goff, D.C., Placebo-controlled trial of glycine added to clozapine in schizophrenia. Am. J. Psychiatry 157 (5), Folstein, M.F., Folstein, S.E., McHugh, P.R., Mini-mental