Expanded Scoring Criteria for the Design Fluency Test: Reliability and Validity in Neuropsychological and College Samples

Transcription

1 Archives of Clinical Neuropsychology, Vol. 14, No. 5, pp , 1999 Copyright 1999 National Academy of Neuropsychology Printed in the USA. All rights reserved /99 $ see front matter PII S (98)00033-X Expanded Scoring Criteria for the Design Fluency Test: Reliability and Validity in Neuropsychological and College Samples Stephanie Lewis Harter and Christine C. Hart Texas Tech University Gregory W. Harter Health South Hospital In spite of sensitivity to right frontal lobe dysfunction, the Design Fluency Test (DFT) has been limited by one global score with little psychometric data. This study developed an expanded scoring system with standardized instructions for multiple dimensions of design performance and provided reliability and validity data in a college (n 64) and diverse neuropsychological sample (n 165). The scoring system allowed reliable scoring of number of novel designs, complexity of designs, variations in designs, and concrete, frankly perseverative, and scribbled responses. Performances for a college sample were relatively stable at 1-month retest, but showed practice effects for number of novel designs and complexity. Two principal components showed modest, expected relationships to other neuropsychological measures in the clinical sample. Clinical subjects with a history of closed-head injury or dementing disorder showed impaired DFT performances, with complexity the most sensitive indicator of impairment National Academy of Neuropsychology. Published by Elsevier Science Ltd Jones-Gotman and Milner s (1977) Design Fluency Test (DFT) showed initial potential for specific sensitivity to right frontal lobe lesions, which often elude neuropsychological tests (cf., Ruff, Light, & Evans, 1987; Stuss & Benson, 1984). Their original study found patients with primarily right frontal or right frontal-central cortical excisions to be most impaired in design fluency, followed by those with right temporal or left frontal excisions. All groups of patients with unilateral cortical excisions were impaired relative to matched, normal controls, except for a group with small left posterior lesions. A subsequent study, including larger groups of patients with more focal lesions, suggested that impairment on the DFT was specific to patients with excisions involving the right frontal lobes, and further suggested that the right central region was particularly important in design fluency. In contrast, those with left central excisions showed the most severe impairments in verbal fluency (Jones-Gotman, 1991a, 1991b). Subsequent studies using the Address correspondence to Stephanie Lewis Harter, Department of Psychology, Box 42051, Texas Tech University, Lubbock, TX ; steph@ttu.edu 419

2 420 S. L. Harter, C. C. Hart, and G. W. Harter DFT have also demonstrated impairments in patients with presumed bilateral frontal lobe dysfunction, including those with closed-head injury (Varney et al., 1996), schizophrenia (Kolb & Whishaw, 1983; Stolar, Berenbaum, Banich, & Barch, 1994), and Parkinson s Disease (Taylor, Saint-Cyr, & Lang, 1986). Design fluency impairments have also been observed in those with Alzheimer s, multi-infarct, and alcohol-related dementia (Bigler, 1995; Bigler et al., 1988). Further, decreases in design fluency, related to impairments in verbal fluency and other frontal lobe-dependent tasks, have been observed with normal aging (Axelrod, 1989; Daigneault, Braun, & Whitaker, 1992; Mittenberg, Seidenberg, O Leary, & DiGiulio, 1989). In spite of this initial promise, the DFT has been criticized for overly vague, global scoring procedures and a lack of psychometric data (Ruff, Evans, & Marshall, 1986). Individual differences in approach to the task, including complex or simplified designs and completely unique or slightly varied designs (Hanks, Allen, Ricker, & Deshpande, 1996; Ruff et al., 1987), have also led to difficulties in interpreting the meaning of individual global scores for number of designs produced. Perhaps because of such reservations, the DFT has failed to achieve the widespread use of analogous verbal fluency measures, particularly in clinical applications (Varney et al., 1996). One strategy to improve the reliability of the DFT has been to increase structure in the nature of the task (e.g., Ruff et al., 1987). However, the unstructured nature of the DFT, while complicating scoring, may increase sensitivity to frontal-lobe mediated executive functions by increasing the initiation and organization required of the patient. The original DFT procedures specify both an unstructured free condition and a more structured fixed condition, in which exactly four lines must be used for each design. The free condition of the DFT appears generally more sensitive than the fixed, although the fixed may be particularly sensitive to Binswanger s disease (Lezak, 1995; Jones-Gotman, cited in Spreen & Strauss, 1998). Previous studies have generally provided normative and reliability data for only one global score from the DFT, representing number of different, nonnameable novel designs produced in 5 minutes. However, researchers have also observed qualitative differences in the performances of groups based on site of and etiology of lesion. Jones-Gotman and Milner found patients with right and right frontal-central lobe excisions to have the highest percentages of perseverative responses, scoring repetitions of similar designs, variations on a theme, and scribbled responses as perseverations (Jones-Gotman, 1991a, 1991b; Jones-Gotman & Milner, 1977). Bigler (1995) also noted increased perseverative errors among demented patients. Increases in perseverative responses, without a reduction in novel design production, have also been noted with aging (Daigneault et al., 1992). Jones-Gotman and Milner (1977) found nameable designs (e.g., pictures of concrete objects) to be associated with right temporal and to a lessor extent right frontal lobe excisions. Previous indications that the more unstructured condition of the DFT is the more sensitive, suggested that increasing standardization in scoring criteria, rather than in the task itself, would best increase reliability while preserving the sensitivity of the test. Thus, this study attempted to increase structure, specificity, and multidimensionality in the scoring procedures for the DFT while preserving the unstructured nature of the task. Extensions and modifications of the original scoring system were developed in order to allow measurement of complexity of designs; differentiate between variations on a theme, scored as perseverations by Jones-Gotman and Milner (1977), and more frankly perseverative responses; and allow reliable scoring of more qualitative aspects of design production. Specific instructions were developed for scoring (a) number of novel abstract designs, (b) complexity, (c) variations on a theme, (d) drawing quality, (e) perseverative responses, (f) concrete responses, and (g) random, scribbled responses. The DFT was administered to a college sample and to a diverse clinical sample referred for

3 Design Fluency Test 421 neuropsychological assessment. Scoring dimensions were evaluated for interrater and test-retest reliability, intercorrelations, relation to other neuropsychological measures, and ability to identify expected impairments in clinical groups. In particular, it was predicted that persons with presumed bilateral frontal lobe damage from closed-head injury would have difficulty on the DFT, in relation to the severity of their head injury. Patients with generalized dementia were also expected to perform poorly on the DFT. METHODS Subjects College participants volunteered to participant in research for extra credit in their Introductory Psychology classes. They were given the DFT on two occasions, approximately 4 weeks apart, as well as a brief medical history interview to screen for any history of neurological injury or illness. College participants were primarily female (91%); White, non-hispanic (89%), and right-handed (86%). Their ages ranged from 17 to 58, with an average of 20. Eighty-one percent of college participants reported no history suggesting neurological injury, illness, or other dysfunction. Of the remaining 19%, 3 participants reported a history of striking their head in a motor-vehicle accident (MVA) resulting in loss of consciousness and or posttraumatic amnesia and 8 participants reported a history of some other blow to the head resulting in a loss of consciousness or posttraumatic amnesia. Only 1 of the participants reporting a history of possible head injury reported continued problems related to the injury. These were headaches and related reading comprehension problems. Three other participants reported recent signs of possible neurological dysfunction, of unknown origin. These included seizures, dizziness, and fainting, and memory problems. Clinical participants included 165 neurological and psychiatric patients. Ages ranged from 13 to 84, with a mean of 37. Years of formal education ranged from 2 to 20, with a mean of 11. Fifty-three percent were female, 85% were right-handed, and 91% were White, non-hispanic. These included all referrals to the neuropsychological services of a rural, regional hospital over a 2-year period who received the DFT as a part of the assessment battery. Participants included both inpatients and outpatients and were referred due to suspected cognitive impairments or to personality changes plausibly related to some neurological injury or disease. Diagnostic and history information were drawn from reports by the referring physicians and from extensive medical history and clinical interviews done by the neuropsychologists. Patients often presented with multiple possible etiologies of neuropsychological dysfunction. Head injury appeared to be the primary etiological factor in 37% of the cases. Ninety-five percent of these cases had sustained their head injury more than a year previously. The time since injury ranged from 0 to 507 months, with a mean of 62 and a median of 20. An additional 21% of clinical participants had a history of head injury, although there was another, more primary reason for referral. Other primary diagnostic groups included depressive disorders (12.7%), alcohol and drug abuse (8.5%), schizophrenia (4.8%), neurotoxic exposure (4.2%), bipolar disorder mixed or manic (4.2%), and a variety of other less frequent neurological and psychiatric disorders. Measures Design Fluency Test (DFT). The DFT was administered according to procedures drawn from Jones-Gotman and Milner (1977), using standard written and oral instructions, in-

4 422 S. L. Harter, C. C. Hart, and G. W. Harter cluding acceptable and nonacceptable examples. Participants were asked to draw as many, different designs as possible in a 5-minute period, avoiding scribbling and any drawing that could be named. Protocols were scored in a blind manner by a neuropsychologist or by a doctoral student specializing in clinical neuropsychology using the standardized scoring instructions prepared for this study (available from the authors). Both of these raters scored all protocols from the college sample and the first 40 protocols from the clinical sample to establish interrater reliability (see Results section). In these cases, scores by the neuropsychologist were used in all analyses following the reliability analyses. Other neuropsychological measures. Clinical participants originally took the DFT as a part of a neuropsychological battery. There was variation among participants as to which other measures were administered, depending upon the referral question. At least 80% of patients completed the Wechsler Adult Intelligence Scale-Revised (n 156; Wechsler, 1981), the Memory Assessment Scales (n 95; Williams, 1991), the Controlled Oral Word Association Test (COWAT, n 98; Benton & Hamsher, 1989), the Trailmaking Test (Trails A & B, n 160; Reitan, 1958), the Short Category Test (n 82; Wetzel & Boll, 1987), Finger Oscillation (Tapping R & L, n 147; Reitan & Wolfson, 1993), the Hand Dynamometer (n 132; Reitan & Wolfson, 1993), and the Dichotic Listening Test (n 132; Roberts, Varney, Paulsen & Richardson, 1990). Thus, these measures were compared to performances on the DFT. RESULTS Psychometric Properties of the DFT Variations on a theme, perseverative responses, and scribbled responses occurred too rarely in the college sample to accurately estimate reliability. Other DFT scores were analyzed using repeated measures analysis of variance with rater and time of testing as independent variables. Means and standard deviations of the DFT scoring dimensions are summarized in Table 1. Main effects for time of testing were significant for number of novel designs, F (1, 63) 12.11, p.001, and complexity of the protocol, F (1, 63) 9.07, p.005, reflecting practice effects on both dimensions. No practice effects were TABLE 1 Means (M) and Standard Deviations (SD) of Interrater and Test- Retest Reliability Data for the College Sample (n 64) Initial Testing Retesting Rater 1 Rater 2 Rater 1 Rater 2 Number of novel designs M SD Complexity M SD Number of concrete responses M SD Drawing quality M SD

5 Design Fluency Test 423 noted for concrete responses or drawing quality. Rater effects were noted only for complexity of the protocol, with the less experienced rater tending to score protocols as more complex, F (1, 63) 10.33, p.005. This may have been due to differences in scoring repetitious elements in designs (e.g., zigzag lines and other repeated patterns), which are given limited credit to avoid inflating scores. Reliability estimates were calculated from these analyses of variance using intraclass correlations as discussed by Winer (1971) and Shrout and Fleiss (1979) and are summarized in Table 2. These estimates were very close to reliability estimates calculated using Pearson correlations. As expected, interrater reliabilities are higher than test-retest reliabilities, since they do not involve changes due to participant s state at the time of testing (e.g., fatigue, motivation). Interrater consistency and test-retest reliabilities for drawing quality are less than desirable, but fair for a rating task requiring subjective clinical judgement (cf., Cicchetti & Sparrow, 1981). All other interrater consistencies are at least in the range recommended for research use by Nunnally (1978), as are test-retest reliabilities for complexity of the protocol and number of concrete responses. Test-retest reliability for number of designs closely approaches the recommended range of r.70. Interrater consistencies for number of novel designs, complexity of the protocol, perseverative responses, and scribbled responses are particularly strong, achieving desirable standards for tests intended for clinical application (Nunnally, 1978). Distributions of DFT scores in the college and neuropsychological samples are summarized in Table 3. No variations on a theme or perseverative responses were noted in the college sample. Concrete and scribbled responses occurred only rarely in the college sample, with only 6 cases offering 1 or more concrete responses (9.3%) and only 3 cases offering a scribbled response (4.7%). Variations on a theme and perseverative, concrete, and scribbled responses were few, but less infrequent, in the neuropsychological sample, appearing in 25.5%, 4.2%, 32.7%, and 15.8% of protocols, respectively. Numbers of perseverations and scribbled responses deviated markedly from normal distribution in the clinical sample. Examination of participants scores on these variables TABLE 2 Reliability of the Expanded Scoring System for the Design Fluency Test Scoring Dimensions Intraclass Correlations Interrater Test-Retest (4 weeks) College students (n 64) Number of novel designs Complexity Number of concrete responses Drawing quality Clinical subjects (n 40) Number of novel designs.98 Complexity.98 Number of variations.77 Number of preseverations 1.00 Number of concrete responses.81 Number of scribbles 1.00 Drawing quality.46 Note. Reliability for variations, perseverations, and scribbles could not be computed for the college student sample, due to the rare occurrence of those responses in that sample. Retest measures were not available for the clinical sample.

6 424 S. L. Harter, C. C. Hart, and G. W. Harter revealed three participants with extreme responses that contributed to the nonnormality of distribution on these scores. One participant with a long history of chronic undifferentiated schizophrenia with generalized impairments on neurocognitive testing produced 206 simple, scribbled responses that also resulted in scoring of 205 perseverative responses, since the responses were virtually identical to each other. Another subject with mild to moderate mental retardation produced 16 perseverative responses. A third participant with a comorbid history of personality disorder, alcohol abuse, and learning disability and a remote history of a mild head injury produced 17 scribbled responses. In order to prevent these outliers from unduly influencing analyses, their scores were transformed to reduce the extremity of responses while preserving rank order, as suggested by Tabachnick and Fidell (1989). Thus, 16 perseverations was transformed to 4, one higher than the next highest number of 3 perseverative responses, and 205 perseverations was transformed to 5. Seventeen scribbled responses was transformed to 7, one higher than the next highest number of 6 scribbled responses, and 206 scribbled responses was transformed to 8. As indicated in Table 3, these transformations increased the normality of distribution for these variables. Group comparisons of diagnostic groups discussed below did not include these three participants because of their comorbid diagnoses and or because of insufficient participants sharing their primary diagnosis. Intercorrelations of DFT scoring dimensions in both the college and clinical samples suggested that it is a multidimensional task. Principal components analyses with oblique rotations were used to summarize the major dimensions reflected in scores. In the college sample, perseverative responses and variations on a theme were not included in analyses since they did not occur on the initial testing. Analyses of the other scoring dimensions yielded 2 principal components with eigenvalues greater than 1.00 (1.79 and 1.16, respectively) accounting for 59% of the variance. Results of the analyses are sum- TABLE 3 Distribution of Design Fluency Test Measures Scoring Dimension M SD Skew Kurtosis College sample (n 64) a Number of novel designs Complexity Number of variations 0 0 Number of perseverations 0 0 Number of concrete responses Number of scribbled responses Drawing quality b Clinical sample (n 165) Number of novel designs Complexity Number of variations Number of perseverations (with transformations) c (.08) (.51) (7.48) (60.02) Number of concrete responses Number of scribbled responses (With transformations) c (.33) (1.14) (4.39) (20.87) Drawing Quality b a Initial testing, neuropsychologist s scores. b Ratings of drawing quality: 0 scribbles only, 1 crude, 2 average, 3 excellent, precise. c Scores on two extreme outliers for each of these variables were reduced to 1 and 2 above the next highest scores, preserving the order of scores.

7 Design Fluency Test 425 marized in Table 4. Correlations of variables with factors suggest that the first component reflects quantity of abstract design production, reflecting number and complexity of novel designs, and the second component reflects scribbled and poorly drawn responses. In the clinical sample, three components were extracted with eigenvalues greater than one (eigenvalues of 2.17, 1.61, and 1.07) accounting for 69% of the variance. The first component appeared to reflect quantity of abstract design production, correlating most highly with number of novel designs (r.83), variations on a theme (r.77), and complexity of the protocol (r.75). The second component appeared to reflect scribbled, perseverative responses, correlating most highly with scribbled responses (r.85) and perseverative responses (r.62) and inversely with drawing quality (r.72). The third component appeared to reflect concreteness. It was most highly correlated with concrete responses (r.89), but was also correlated with perseverative responses (r.59) and was inversely related to complexity (r.42). The third component appeared to be less distinctly and reliably measured, since only number of concrete responses had its highest correlation with this component. Thus, a second principal components analysis of the clinical sample extracted two components, allowing more direct comparison with the structure of scores in the college sample. Results of this analysis are also summarized in Table 4. The first two components accounted for 54% of the variance. Similar to the college sample, the first component appeared to reflect quantity of abstract design production, while the second component appeared to reflect scribbled and perseverative responses. Concrete responses were inversely related to the first component and directly related to the second component. These components had a low inverse relationship in the clinical sample (r 13), but were relatively independent in the college sample (r.06). Relation to Demographic Variables The DFT appeared relatively independent of demographic factors. Men and women did not differ in complexity of the protocol, design quality, or numbers of variations, perseverations, concrete responses, or scribbled responses in either the college or clinical samples. Men and women did not differ in number of novel designs in the clinical sam- TABLE 4 Principal Components Analyses of the Design Fluency Test in College and Clinical Samples Scoring Dimension Structure Matrix Abstract Design Productivity Scribbling/ Perseveration College sample (n 64) Complexity Number of novel designs Number of concrete responses Number of scribbles Drawing quality Clinical Sample (n 165) Number of novel designs Complexity Number of variations Number of perseverations Number of scribbles Drawing quality Number of concrete responses

8 426 S. L. Harter, C. C. Hart, and G. W. Harter ple, but men did produce significantly more designs in the college sample (males: M 15.17, SD 8.70, n 6; females: M 9.14, SD 5.12, n 58; p.05). However, the small number of men in the college sample (few men volunteered for the study) raised questions about the representativeness of that subgroup. Education was unrelated to all scoring dimensions in the clinical sample. There was a nonsignificant trend for more advanced students to produce more novel designs in the college sample (r.23, p.06). Age was unrelated to all scoring dimensions in the college sample and was only slightly related to an increased number of variations on a theme in the clinical sample (r.17, p.05). Hand dominance was related to some of the more qualitative scoring dimensions. Left-handed participants had poorer drawing quality in the clinical sample (right-handed: M 1.62, SD.58; left-handed: M 1.29, SD.55; p.05), more perseverations (right-handed: M.02, SD.19; left-handed: M.62, SD 1.41; p.001), and more scribbled responses (right-handed: M.28, SD.89; left handed: M 2.54, SD.52; p.001). They also tended to have more variations on a theme (right-handed: M.75, SD 1.83; left-handed: M 1.79, SD 4.74; p.06). Left-handed participants in the college sample also tended to have poorer drawing quality (right-handed: M 1.80, SD.52; left-handed: M 1.44, SD.53; p.06). As discussed previously perseverative responses and variations on a theme did not occur in initial testing in the college sample. Relation to Other Neuropsychological Measures Other neuropsychological measures were not available for the college sample. Principal component scores from the two-factor analysis of the clinical sample were compared to performances on other neuropsychological measures completed by at least 80% of the clinical sample (see Methods section). Component 1, abstract design productivity, was weakly related to performance IQ (r.18, p.02) and tended to be related to verbal IQ (r.15, p.06). It had a low relation to verbal fluency on the COWAT (r.23, p.005) and tended to be related to tapping for the right hand (r.16, p.05). It was inversely related errors on the Short Category test (r.18, p.05) and to seconds required to complete Trails A (r.18, p.05) and Trails B (r.24, p.01). Component 2, scribbled, perseverative responses, was inversely related to verbal IQ (r.31, p.001) and performance IQ (r.34, p.001), verbal fluency (r 0.26, p.001), verbal (r.31, p.001) and visual (r.24, p.005) memory on the MAS, dichotic listening performance for the right (r.33, p.001) and left (r.27, p.001) ears, and tapping right (r.30, p.001) and left (r.19, p.05). It was also related to seconds required to complete Trails B (r.34, p.001) and errors on the Short Category test (r.25, p.005). These relationships to other neurocognitive measures were in the expected directions, but their modest magnitude suggested that the DFT is not redundant to other measures in a neuropsychological battery. Discrimination Between Clinical Groups and the College Comparison Group DFT measures were included in a multivariate analysis of variance (MANOVA), comparing various clinical diagnostic groups to the nonclinical, college student comparison group. The 12 students reporting a possible history of neurological injury or other dysfunction were excluded from the college group. Six patients with a primary diagnosis of minor closed head injury, involving no loss of consciousness (LOC) or posttraumatic amnesia (PTA) were excluded from analyses, due to inadequate group size. The remaining participants with a primary diagnosis of closed-head injury were divided into mild to

9 Design Fluency Test 427 moderate (LOC and PTA less than 2 hours) and severe (LOC and PTA greater than 2 hours) groups. Additional clinical groups included those with primary diagnoses of a dementing disorder, alcohol or drug abuse or dependency, and unipolar depressive disorder. Other diagnostic categories were excluded from analyses due to small subsample size. Dependent variables included number of novel designs, complexity, drawing quality, variations on a theme, and perseverative, scribbled, and concrete responses. See Table 5 for a summary of DFT performances for each diagnostic group. Diagnostic groups differed significantly in multivariate comparison, F (Wilks criterion, 35, 570) 2.78, p.001. Univariate comparisons were significant for complexity, concrete responses, scribbled responses, number of novel designs, and variations on a theme, in order of decreasing effect size (.18,.11,.08, and.08, respectively). Planned comparisons of each diagnostic group to the college comparison group (with significance level at p.05) on variables significant in univariate analyses indicated that the mild to moderate head injury group had more scribbled responses and variations on a theme than the college group. The severe head injury group had less complex protocols with more concrete responses than the college group. The dementing disorders group had less complex protocols and more scribbled responses and variations than the college group. The depressive disorders group produced more novel designs, but less complex protocols with more concrete responses than the college group. They also had more variations on a theme. Planned comparisons of the mild-moderate and severe head injury groups and dementing disorders group to other clinical groups on variables with overall TABLE 5 Means (M) and Standard Deviations (SD) of Diagnostic Groups and a Nonclinical College Student Comparison Group on Dimensions of Design Fluency Test Performance Diagnostic Group Number of Novel Designs Complexity Drawing Quality Variations Perseverations Concrete Responses Scribbled Responses Mild moderately head injured (n 35) M SD Severely head injured (n 20) M SD Dementia (n 9) M SD Etoh/drug abuse (n 14) M SD Unipolar depression (n 17) M SD Nonclinical college (n 52) M SD

10 428 S. L. Harter, C. C. Hart, and G. W. Harter univariate significance were also conducted. The mild to moderate head injury group had significantly more complex protocols than the severe head injury group. They had more scribbled responses and fewer novel designs than the depressive disorders group. They also had more scribbled designs than the alcohol drug abuse group. In addition to the previously mentioned differences from the mild-moderate head injury group, the severe head injury group had fewer novel designs than the depressive disorders group. They also had less complex protocols and more concrete responses than the alcohol drug abuse group. The dementing disorders group did not differ significantly from any of the other clinical groups on number of novel designs, complexity, variations, concrete responses, or scribbled responses. However, the small size of the dementing disorders group limited power for these comparisons. Impaired Performances For DFT dimensions that significantly discriminated diagnostic groups in the above analyses, percent of participants in each diagnostic group with impaired performances was calculated to further assess clinical relevance of the DFT scoring dimensions. Number of novel designs and complexity were considered impaired if the participant performed more than 1.5 standard deviations below the college comparison group on that variable (1 or no novel designs, 56 or less for complexity of the protocol). Variations on a theme, scribbled responses, and concrete responses rarely or never occurred in the college comparison samples. Thus, any response of this type was considered to indicate possible impairment. Percentages of each diagnostic group exhibiting potentially impaired performances are summarized in Table 6. Of the 52 college comparison participants, only 10 (19.2%) had a potential impairment on any dimension of the DFT. These included 4 with reduced complexity (7.7%), 4 with 1 to 3 concrete designs (7.7%), and 3 with 1 scribbled response (5.8%). All of the participants with dementing disorders had some impaired performance. From 63 to 65% of the other diagnostic groups had some impaired performance. Reduced complexity and concrete responses were particularly characteristic of the severe head injury group and the dementing disorders group. The dementing disorders group was also characterized by variations on a theme and scribbled responses. However, variations on a theme Diagnostic Group TABLE 6 Percent of Each Diagnostic Group with Impaired Performances Number of Novel Designs (%) Complexity (%) Variations (%) Concrete Responses (%) Scribbled Responses (%) Any Impairment (%) Mild moderately head injured (n 35) Severely head injured (n 20) Dementia (n 9) Etoh/drug abuse (n 14) Unipolar depression (n 17) Nonclinical college (n 52)

11 Design Fluency Test 429 and scribbled responses were not reliably associated with severity of head injury, since they were more likely to occur in the mild to moderate than the severe head injury groups. DISCUSSION Results provide initial reliability and validity data for an expanded scoring system for the DFT. Performances remained relatively stable on 1-month retesting, especially concrete responses (r.91) and complexity of the protocol (r.77), filling a previous need for published test-retest reliability estimates. There were modest practice effects for complexity of the protocol, which increased by 11%, and for novel designs, which increased by 22% (two designs). Previous studies have reported interrater reliabilities for number of novel designs ranging from.64 to.87 (Jones-Gotman, 1991a; Ross, Axelrod, Hanks, Kotasek, & Whitman, cited in Spreen & Strauss, 1998; Woodard, Axelrod, & Henry, 1992), in contrast to the current reliabilities of 1.00 in the college sample and.98 in the clinical sample. The expanded instructions also allow reliable scoring of the complexity of designs in the protocol (r.97.98), providing more comprehensive and sensitive measurement of design production. The expanded scoring instructions partition rule violations into perseverations, concrete responses, and scribbled responses. The reliability of rule violations has rarely been addressed in previous studies, since in the original scoring procedures rule violations were subtracted from total number of designs to produce one global score for number of novel designs. Two studies giving reliabilities for rule violations have reported inter-rater reliabilities of.21 to.74, depending upon the nature of the violation (Ross et al., cited in Spreen & Strauss, 1998; Woodard et al., 1992), in contrast to present reliabilities of.81 to Previous scoring systems have not allowed separate examination of variations on a theme, which were combined with frank perseverations. The present scoring system allows variations to be scored reliably enough (r.77) to allow investigation of whether such responses represent perseverative repetitions or strategic variations to increase design production (cf., Vik & Ruff, 1988). Attempts to provide qualitative ratings of the skill and precision of drawing yielded lower reliabilities than the more quantitative scoring dimensions. Performance on the DFT appeared to be multidimensional in both college and clinical samples. The primary and secondary components in both samples included abstract design productivity and scribbled responses. There also appeared to be a smaller third component in the clinical sample reflecting concrete responses that did not emerge in the college sample due to the reduced number of scoring dimension relevant in that sample. Modest inverse relationships were found between scribbled, perseverative responses and verbal and performance IQ, verbal fluency, and other neuropsychological tests dependent upon integrity of the frontal and temporal lobes. Generally smaller relationships were observed between abstract design productivity and performance IQ, verbal fluency, and other measures related to frontal lobe functioning. The pattern and magnitude of these relationships is consistent with previous findings for number of novel designs (Varney et al., 1996) in suggesting that Design Fluency is not a redundant measure in a neuropsychological battery, that it is related in the expected manner to other tests of frontal lobe functioning, and that poor performance is not simply a matter of impaired motor speed or visual-motor integration. Results are consistent with previous research (Varney et al., 1996) in suggesting impaired DFT performance in those with a history of closed-head injury. The expanded

12 430 S. L. Harter, C. C. Hart, and G. W. Harter scoring system allows more complete assessment of dimensions of impairment. Reduced complexity of responses in the protocol and concrete responses were particularly related to severity of head injury. Results are also consistent with previous research in suggesting impaired DFT performance in patients with generalized cognitive deficits associated with dementing disorders (Bigler, 1995; Bigler et al., 1988). All dementing patients were impaired on some dimension of the DFT. Performances of dementing patients were particularly characterized by reduced complexity and increased scribbled responses and variations on a theme in contrast to the college comparison sample. High percentages of this sample also showed concrete responses. Overall, complexity of the protocol appeared to be the most sensitive indicator of abstract design productivity, more effectively discriminating groups with presumed frontal lobe impairment and generalized dementia. However, the present failure of number of novel designs to discriminate head-injured and dementing groups from the college comparison sample may be partially due to characteristics of the college sample. College students had the highest average complexity of designs, which necessarily limited the number of designs that they could produce within the time limit. Number of novel designs in the college comparison group was lower than that of older normal controls in previous studies (Jones-Gotman & Milner, 1977; Varney et al., 1996; Woodard et al., 1992), including a normal aging sample (Axelrod, 1989). The complexity of designs in those samples is not known. However, the current college sample did not differ in number of novel designs and complexity from a larger, gender-balanced college sample used in a study of test order effects (Hart, Harter, Miller, & Harter, 1996). College students may be differently motivated than older adult volunteers and more inclined to produce complex designs. The current college sample was also predominately female, unlike the clinical sample. Males in the college sample produced more designs than females. However, there were no gender differences for DFT performances in the present clinical sample, in the previous college sample (Hart et al., 1996), or in Varney et al. (1996) control and clinical samples. Overall, comparison to previous studies suggests that use of a college comparison sample may have yielded an underestimate of impairments in number of novel designs for the clinical groups. However, further comparisons of normal groups on the DFT would be necessary to confirm this speculation. In contrast to the college comparison group, the depressive disorders group produced a higher number of novel designs, more similar to comparison groups used in previous studies. Both head-injured groups in the current study produced fewer novel designs than the depressive disorder s group. All clinical subjects in the current study were referred due to suspected cognitive problems or complaints or personality changes subsequent to neurological insult. Thus, it is unclear that the depressive disorders or the alcohol and drug abuse groups were free of neurological impairment. In addition to lack of an age-matched normal comparison sample, the current study is also limited by relatively small size of some clinical groups. For instance, the small number of patients with dementing disorders limited power for comparisons to other groups and precluded examination of different types of dementia. Also, the very small number of patients with focal or clearly lateralized lesions prevented assessment of sensitivity of the DFT to laterality. The expanded scoring system offers the opportunity to further study qualitative differences in DFT performance related to locus and etiology of neurological injury. The current patterns of performance for head-injured versus demented patients are suggestive in this regard and should be cross-validated with a larger sample. It would additionally be useful in future studies to investigate the specificity of the DFT to neurological

13 Design Fluency Test 431 disorders, in comparison to psychiatric comparison groups that do not include patients with suspected neurocognitive impairment. REFERENCES Axelrod, B. N. (1989). Frontal lobe functioning in normal aging. Unpublished doctoral dissertation, Wayne State University, Detroit, MI. Benton, A. L., & Hamsher, K. de S. (1989). Multilingual Aphasia Examination. Iowa City, IA: AJA Associates. Bigler, E. D. (1995). Design fluency in dementia of Alzheimer s type, multi-infarct dementia and dementia associated with alcoholism. Applied Neuropsychology, 2, Bigler, E. D., Schultz, R., Grant, M., Knight, G., Lucas, J., Roman, M., Hall, S., & Sullivan, M. (1988). Design fluency in dementia of the Alzheimer s type: Preliminary findings. Neuropsychology, 2, Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, Daigneault, S., Braun, C. M. J., & Whitaker, H. A. (1992). Early effects of normal aging on perseverative and non-perseverative prefrontal measures. Developmental Neuropsychology, 8, Hanks, R. A., Allen, J. B., Ricker, J. H., & Deshpande, S. A. (1996). Normative data on a measure of design fluency: The Make a Figure Test. Assessment, 3, Hart, C. C., Harter, S. L., Miller, S. A., & Harter, G. W. (1996). Design Fluency within a cognitive battery: A study of test order effects. Paper presented at the National Academy of Neuropsychology, 16th Annual Meeting, New Orleans, LA. Jones-Gotman, M. (1991a). Localization of lesions by neuropsychological testing. Epilepsia, 32, Jones-Gotman, M. (1991b). Presurgical psychological assessment in children: Special tests. Journal of Epilepsy, 3 (Suppl. 1), Jones-Gotman, M., & Milner, B. (1977). Design Fluency: The invention of nonsense drawings after focal cortical lesions. Neuropsychologia, 15, Kolb, B., & Whishaw, I. Q. (1983). Performance of schizophrenic patients on tests sensitive to left or right frontal, temporal, or parietal function in neurological patients. Journal of Nervous and Mental Disease, 171, Lezak, M. D. (1995). Neuropsychological assessment. New York: Oxford University Press. Mittenberg, W., Seidenberg, M., O Leary, D. S., & DiGiulio, D. V. (1989). Changes in cerebral functioning associated with normal aging. Journal of Clinical and Experimental Neuropsychology, 11, Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill. Reitan, R. M. (1958). Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, Reitan, R. M., & Wolfson, D. (1993). The Halstead-Reitan Neuropsychological Test Battery: Theory and clinical interpretation. Tucson, AZ: Neuropsychology Press. Roberts, R., Varney, N. R., Paulsen, J., & Richardson, E. (1990). Dichotic listening and partial seizures. Journal of Clinical and Experimental Neuropsychology, 12, Ruff, R. M., Evans, R., & Marshall, L. F. (1986). Impaired verbal and figural fluency after head injury. Archives of Clinical Neuropsychology, 1, Ruff, R. M., Light, R. H., & Evans, R. W. (1987). The Ruff Figural Fluency Test: A normative study with adults. Developmental Neuropsychology, 3, Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, Spreen, D., & Strauss, E. (1998). A compendium of neuropsychological tests (2nd ed.). New York: Oxford University Press. Stolar, N., Berenbaum, H., Banich, M. T., & Barch, D. (1994). Neuropsychological correlates of alogia and affective flattening in schizophrenia. Biological Psychiatry, 35, Stuss, D. T., & Benson, D. F. (1984). Neuropsychological studies of the frontal lobes. Psychological Bulletin, 95, Tabachnick, B. G., & Fidell, L. S. (1989). Using multivariate statistics (2nd ed.). New York: Harper Collins. Taylor, A. E., Saint-Cyr, J. A., & Lang, A. E. (1986). Frontal lobe dysfunction in Parkinson s disease: The cortical focus of neostriatal outflow. Brain, 109, Varney, N. R., Roberts, R. J., Struchen, M. A., Hanson, T. V., Franzen, K. M., & Connell, S. K. (1996). Design

14 432 S. L. Harter, C. C. Hart, and G. W. Harter Fluency among normals and patients with closed head injury. Archives of Clinical Neuropsychology, 11, Vik, P., & Ruff, R. R. (1988). Children s figural fluency performance: Development of strategy use. Developmental Neuropsychology, 4, Wechsler, D. (1981). WAIS-R Manual. New York: The Psychological Corporation. Wetzel, L., & Boll, T. J. (1987). Short Category Test, Booklet Format. Los Angeles, CA: Western Psychological Services. Williams, J. M. (1991). Memory Assessment Scales. Odessa, FL: Psychological Assessment Resources. Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw Hill. Woodard, J. L., Axelrod, B. N., & Henry, R. R. (1992). Interrater reliability of scoring parameters for the Design Fluency Test. Neuropsychology, 6,