The reliability of production strategy scores for the Ruff Figural Fluency Test

Transcription

1 Archives of Clinical Neuropsychology 18 (2003) The reliability of production strategy scores for the Ruff Figural Fluency Test Thomas P. Ross, E. Lindsay Foard, F. Berry Hiott, Angela Vincent Department of Psychology, College of Charleston, 66 George Street, Charleston, SC 29424, USA Accepted 14 June 2002 Abstract This study examined the interrater and test retest reliability of qualitative scoring indices for the Ruff Figural Fluency Test (RFFT) in a sample (N = 90) of healthy college undergraduates. In addition to rotational and enumerative strategies proposed by Ruff (1988), other supplemental indices were developed and examined. Seven raters scored protocols independently using guidelines proposed by Ruff (1988) and additional scoring examples developed by the primary investigator. Intraclass correlation coefficients (ICCs) were excellent, ranging from r ICC =.91 for number of strategic clusters, to r ICC =.79 for mean cluster size. Coefficients of stability (n = 48; mean interval = 7 weeks) for strategy scores were also acceptable (e.g., r ICC =.80 for enumerative, r ICC =.71 for rotational). This study is the first to examine the psychometric properties of these supplemental scores for the RFFT. Reliability coefficients obtained for traditional RFFT indices (e.g., unique designs, perseverations) were consistent with previous investigations (e.g., Basso, Bornstein, & Lang, 1999; Ruff, 1988). Modest practice effects were observed on unique designs produced, but not for strategy scores. The implications of using RFFT strategy scores for assessing executive functions are discussed and considerations for future research are presented National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. Keywords: Ruff Figural Fluency Test; Design fluency; Interrater; Test retest; Executive functioning; Strategy use; Qualitative scoring The Ruff Figural Fluency Test (RFFT) was designed by Ruff (1988) to be a nonverbal analogue of verbal fluency measures (e.g., Controlled Oral Word Association Test; Benton, Hamsher, & Sivan, 1983), whereby subjects must generate as many unique responses as possible while trying their best to avoid repetitions. Similar to measures of verbal fluency, this task is often useful for assessing the perseverative tendencies of patients with frontal lobe lesions Corresponding author. Tel.: ; fax: address: rosstp@cofc.edu (T.P. Ross) /$ see front matter 2002 National Academy of Neuropsychology. PII: S (02)

2 880 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) (see Stuss & Benson, 1986). Patients with lesions to the frontal lobes and associated brain areas produce very few unique designs overall, and exhibit a tendency to reproduce items they have drawn previously (see Lezak, 1995; Ruff, Allen, Farrow, Niemann, & Wylie, 1994; Spreen & Strauss, 1998). The RFFT procedure requires participants to generate as many line drawings as possible across five trials (i.e., parts), each lasting 60 s. Participants are presented test forms that contain rows of squares and within each is an array of five dots. Participants are then informed that, within each square, they must connect two or more dots by always using straight lines. The goal of the task is to generate as many unique designs as possible without replicating any drawings (Ruff, 1988). The ability to initiate and sustain mental productivity, and the ability to self-monitor and regulate responding are crucial for effective performance. Because of these task demands, the RFFT is considered a measure of executive functioning (Denckla, 1994; Lezak, 1995; Spreen & Strauss, 1998). The RFFT is scored by counting the number of total designs produced and the number of repetitious (i.e., perseverative) designs. The number of unique designs is calculated by subtracting the number of perseverative designs from the total designs produced. Additionally, an error ratio is calculated by dividing the total number of perseverative designs by the number of unique designs. In addition to these standard scoring procedures, examiners can calculate a qualitative scoring index called a production strategy score. Production strategy scores are defined as three or more consecutive designs for which the subject has simply rotated a drawing to make it different or has enumerated upon a general design principal (Ruff, 1988). The generation and implementation of efficient production strategies is an important cognitive element assessed by the RFFT (Lezak, 1995; Ruff, 1988; Vik & Ruff, 1988) and similar tasks (e.g., the Five Point Test; Regard, Strauss, & Knapp, 1982). The use of production strategies can be an effective means to comply simultaneously with two key requirements: optimizing novel output while minimizing repetitious responding (Lezak, 1995). According to Lezak (1995): Generally, the greatest productivity with the fewest perseverations is achieved by persons who quickly develop and then maintain a strategy so that each square no longer calls for a unique solution but rather, the pattern for a long series of squares has been predetermined by the strategy. (p. 670) Ruff (1988) contended that analysis of production strategies can elucidate whether a patient s deficient RFFT performance reflects problems with initiation versus impaired planning abilities. Analyses of cases studies suggest that patients with initiation problems tend to produce very few designs overall. In contrast, patients with deficits in planning produce more designs, but have done so at the expense of more perseverative responses and fewer production strategies (Ruff, 1988). The RFFT is sensitive to cerebral dysfunction and to right anterior brain lesions in particular (e.g., Baldo, Shimamura, Delis, Kramer, & Kaplan, 2001; Ruff, 1988; Ruff, Allen, Farrow, Niemann, & Wylie, 1994). It has been used to research a variety of populations including healthy persons (e.g., Basso, Bornstein, & Lang, 1999; Demakis, 1999; Ruff, Light, & Evans, 1987), and individuals with brain injury (e.g., Baldo et al., 2001; Ruff et al., 1994), dementia (e.g., Fama et al., 1999), and psychiatric illness (e.g., Jolly, 2000).

3 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Although the RFFT has been used widely in research to assess executive functioning, there is a dearth of studies examining the psychometric properties of this measure. The few studies that exist are restricted to reports of the standard scoring indices (e.g., number of unique designs, number of perseverations) without mention of production strategy scores. Only two studies have examined the interrater reliability of the RFFT. Berning, Weed, and Aloia (1998) examined the interrater reliability of the RFFT in a sample of 124 college undergraduates (median age = 20). Their investigation, however, was not typical of traditional interrater reliability examinations. Berning and colleagues employed 30 pairs of naive raters (i.e., college freshman) who each scored 5 10 RFFT protocols after being administered the RFFT. Intraclass correlations between pairs of RFFT protocol scores were r ICC =.93 for total unique designs, r ICC =.74 for total perseverations, and r ICC =.66 for error ratio scores. An interesting finding of Berning et al. s (1998) study was that raters with additional practice (i.e., having completed 10 versus 5 practice protocols) produced significantly higher interrater reliability coefficients. A second study by Sands (1998) examined the interrater reliability of the RFFT in a sample of 50 patients with mixed neurologic disease. Using two raters only, Sands (1998) reported coefficient values of r ICC =.99 for unique designs and r ICC =.99 for perseverative errors. Basso et al., 1999 examined the temporal stability of the RFFT in a sample of 50 healthy men (mean age = 32; mean education = 14.5 years) across a 12-month testing interval. They reported test retest reliability coefficients of r =.71 for total unique designs and r =.39 for the error ratio score. Basso et al. also reported modest, but significant, practice effects for the number of unique designs, whereby participants mean output increased by 6.81 designs. No significant practice effects were observed for the error ratio score. Ruff (1988) reported similar stability coefficients over a 6-month interval in a sample of 95 healthy adults (r =.76 for unique designs and r =.36 for total perseverations). The mean age and education of the retest sample (n = 95) are not reported in the RFFT manual, however Ruff (1988) noted that they did not differ from the standardization sample. Ruff (1988) also observed an average increase of eight unique designs upon retesting. Demakis (1999) reported a stability coefficient of r =.88 for RFFT total unique designs over a 3-week interval in a sample of 21 college undergraduates (mean age = 22.5 years; mean education = 13.6 years) who functioned as the control group in an analogue investigation of serial malingering. He also reported very large practice effects, whereby participants generated an average of 17 more unique designs upon retesting. The larger practice effects observed in the Demakis (1999) investigation are likely attributable to the shorter time interval between testing sessions, as compared to that employed in Basso et al. s (1999) and Ruff s (1988) investigations. Despite the initial efforts described above, there are currently no studies that examine the basic psychometric properties of RFFT production strategy scores. Additionally, there are no studies that examine the relationship between production strategies and unique and perseverative designs to examine their potential role in effective RFFT performance. This study sought to improve upon these limitations by examining the interrater reliability and temporal stability of the standard and production strategy scores for the RRFT, as well as investigating the correlations among these variables. Based on Lezak s (1995) interpretation of nonverbal fluency performance, it was anticipated that production strategy indices would

4 882 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) relate positively to unique design output, whereas negative associations would result among production strategy indices and perseverative errors. 1. Method 1.1. Participants After review and approval from the institutional review board, undergraduates (N = 95) were recruited from introductory psychology courses at an urban, midwestern university. Persons with a history of neurological disorder, learning disability or psychiatric conditions involving medication usage were excluded from data analyses. Using these criteria, five individuals were excluded (one with reported epilepsy, two due to learning disability and/or ADHD, and two due to psychiatric conditions) resulting in 90 healthy persons for analyses. Individuals were not paid for their participation, but each received credit toward their psychology courses. Of the 90 healthy participants (55% female; 89.8% right handed), 44% described themselves as Caucasian, 30% were African American, 11% were Asian American; 9% were Hispanic, and 6% reported other ethnic backgrounds. Participants were between the ages of 18 and 69 years (M = 23.9; S.D. = 7.3), and the mean estimated Full Scale IQ using the North American Adult Reading Test (NAART; Blair & Spreen, 1989) was (S.D. = 9.2). To ensure an adequate sample size for test retest reliability analyses, each participant was invited to volunteer for a second testing session. Forty-eight participants (53% of the initial sample) were retested (mean interval = 35.2 days; range = days). There were no significant differences between the test retest sample (n = 48) and those who did not retest (n = 42) for age, education, ethnicity, or estimated IQ Materials The RFFT (Ruff, 1988), described above, was used to investigate the interrater reliability and temporal stability of production strategies for nonverbal fluency. This paper and pencil measure consists of five parts that each present a slightly different test stimuli pattern consisting of five dots. Part I displays the dots alone, arranged in a concentric fashion. Parts II and III contain the same patterns of dots, but each includes different distracters. Parts IV and V contain different variations of the original pattern of dots without any distracting stimuli. Subjects are instructed to draw as many unique designs as possible within a set time period, while doing there best not to repeat any designs. A stopwatch is used to monitor the 60 s time limit imposed for each part. The NAART (Blair & Spreen, 1989) was employed to estimate the intellectual level of the sample in a time-efficient manner. Several studies have demonstrated the validity of the NAART for this purpose (see Spreen & Strauss, 1998). This measure requires that persons read aloud through a list of words that cannot be read phonetically, so correct pronunciation of the items requires prior experience and mastery. In addition to estimating intellectual functioning, performance on this measure is closely associated with vocabulary and education level (Blair & Spreen, 1989).

5 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) To assess the health history of the sample, a self-report questionnaire was constructed and administered to all participants. In addition to demographic variables, the questionnaire assessed for a history of neurological disorders, psychiatric illness, and other medical disorders known to affect neuropsychological functioning (e.g., hypothyroidism, diabetes) Procedures and scoring After obtaining their informed consent, participants were first administered the health survey, then the RFFT and NAART. The primary investigator, who was an advanced graduate student in clinical psychology at the time of data collection, completed the majority of the test administration. Two other advanced graduate students assisted with additional data collection. These measures were administered in the context of a larger battery of tests used for the primary author s dissertation project. The RFFT and NAART were administered in accordance with published instructions (Ruff, 1988; Blair & Spreen, 1989). After test administration, all participants were appropriately debriefed. To provide a stringent examination of interrater reliability, seven raters (the primary investigator who is now a PhD level psychologist and six senior psychology majors) scored all RFFT protocols. The six student raters were advanced undergraduate psychology majors who functioned as volunteer research assistants in the primary investigator s neuropsychological laboratory. Each had received prior training in the administration and scoring of several neuropsychological tests. To train the raters to score RFFT production strategies, each was provided with the scoring rules published in the RFFT manual (Ruff, 1988), 20 practice protocols generated for this purpose, and additional scoring examples that were generated by the primary investigator. Review of the literature on the training of raters suggests that interrater consistency is best promoted by providing raters with opportunity for group discussion and with practice exercises (Cone, 1999). Following these recommendations, the practice protocols were scored and discussed during four weekly meetings supervised by the primary investigator. Following these training sessions, each rater scored all protocols independently. With the exception of interrater reliability analyses, the calculation of descriptive data, correlations, and stability coefficients required only one score (i.e., rater) per case. This was done to eliminate variance attributable to scoring differences for these analyses. Therefore, the values generated by the one rater who yielded the most accurate scores were used. In instances where scores between this primary rater and another rater were discrepant, the protocol was carefully reinspected by the primary investigator for accuracy in order to obtain the correct values for data analysis. The seven raters scored the RFFT protocols following procedures outlined in pages 4 7 of the professional manual (Ruff, 1988). This study examined the standard RFFT scoring indices as well as the qualitative indices. Standard RFFT indices include the number of unique designs generated, the number of perseverations, and an error ratio (i.e., the number of perseverative errors divided by the number of unique designs). The qualitative scores, known as production strategies, are defined as three or more consecutive designs for which the subject has employed a systematic process to generate designs. They are categorized into two types: a rotational strategy that involves simply rotating a drawing systematically clockwise or counterclockwise within the array of dots to make it appear differently in each square, or an enumerative strategy

6 884 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) whereby subjects enumerate upon a design by systematically adding or removing one line (see Ruff, 1988, pp. 4 7). Although infrequent in the present sample, perseverative responses were included among design sequences when scoring production strategies. This was done because the intent of production strategy scoring is to assess planning and organization, rather than self-monitoring per se. Moreover, the RFFT already includes indices that better reflect self-monitoring such as the total number of perseverations and the error ratio (Ruff, 1988). The one exception was that a perseverative design was not included as part of a strategic cluster if the drawing already appeared in that same cluster or sequence of designs. One limitation of the RFFT manual is that it provides illustrated examples of production strategies using test stimuli for Parts I and II only. For Parts IV and V, the arrangement of the dots is quite different from the more concentric arrangement for Parts I and II, therefore strategies (rotational in particular) may be less recognizable to raters. To assist raters and in turn promote higher interrater reliability, additional examples were generated to illustrate the scoring principles for Parts IV and V. Rotational and enumerative strategies were tallied separately, but the total number of production strategies or strategic clusters was also counted. The term strategic cluster refers to groups of consecutive designs that utilized either a rotational or enumerative principle. Production strategies for Part III of the test were not scored, as the nature of the stimuli confound the test-takers ability to freely utilize rotational and enumerative strategies. Part III of the RFFT was designed to be more difficult by imposing distracting stimuli (i.e., extraneous lines) and by reducing the amount of lines the test-taker may use to connect the dots as two such lines are already drawn (Ruff, 1988). In addition to the production strategy scores proposed by Ruff (1988), two other indices of production strategy use were developed and evaluated. The mean cluster size and the percentage of designs used in strategies. Cluster sizes were conceptualized to reflect the extent to which a participant applied a single strategy across consecutive designs. This index reflects whether a test-taker employed a production strategy to the minimal or maximum extent possible. A cluster size of 1 reflects the minimum amount of consecutive designs that constituted a production strategy, which by Ruff s (1988) definition, is three designs. A cluster size of 2 reflects the use of the minimum number of designs (3) plus 1 for a total of 4 consecutive designs, and a cluster size of 3 reflects the use of the minimum number of designs (3) plus 2 for a total of five consecutive designs, etc. The mean cluster size across the four RFFT parts (I, II, IV and V) was calculated to assess the extent to which test-takers employed more or less elaborate clusters, on average. The percentage of designs used in strategies index was calculated by counting the number of designs incorporated into production strategies and dividing by the number of designs overall (including perseverations). Similar in principle to the RFFT error ratio, this index takes the participants overall fluency into account when expressing participants use of production strategies. To illustrate how this index is interpreted, consider two hypothetical protocols. Person A may generate a similar number of production strategies (e.g., 8 clusters) as person B (e.g., 7 clusters); however, 65% of person A s designs were incorporated into production strategies, whereas person B incorporated only 23%. Person A therefore utilized production strategies to a much greater extent than person B, a difference that would be obscured if only the total number of strategies were examined.

7 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Table 1 Mean and standard deviations for RFFT indices for initial (N = 90) and retest (n = 48) samples Mean (S.D.) Initial sample Retest sample Unique designs (23.1) (24.6) Total perseverations 8.4 (8.8) 8.2 (8.1) Error ratio.0790 (.0521).0753 (.0695) Rotational strategies 2.68 (2.60) 2.54 (2.41) Enumerative strategies 1.42 (2.33) 1.81 (3.12) Total strategic clusters 3.87 (3.43) 4.24 (4.51) Mean cluster size 1.73 (1.14) 1.78 (1.39) Percent designs in strategies 17.2 (15.6) 17.0 (15.4) 2. Results The means and standard deviations for all RFFT scores for initial (N = 90) and retest samples (n = 48) are shown in Table 1. No gender differences were observed on any indices for either sample (P >.05); therefore data for males and females are not presented separately. Consistent with previous research (e.g., Basso et al., 1999; Ruff, 1988), the average performance of participants improved upon retesting. For those participants (n = 48) who were retested, paired sample t-tests of time 1 versus time 2 scores revealed a modest, but significant increase in performance for the number of unique designs produced [t (47) = 7.02; P <.01]. On average, participants generated eight more unique designs at time 1 (M = 106.1; S.D. = 22.2) as compared to time 2 (M = 114.5; S.D. = 24.6). No significant differences for time 1 versus time 2 scores were observed for perseverative responses, the error ratio, or any production strategy indices. The higher number of RFFT unique designs observed in the retest sample at time 2 suggests a practice effect. Because the retest group was not selected randomly, RFFT scores at time 1 were examined for preexisting differences between the 48 persons who retested and those 42 individuals tested at time 1 only. There were no significant between-group differences for any RFFT scores at time 1 (P >.05). Therefore, the gains in total unique designs in the retest sample could not be attributed to superior nonverbal fluency at time 1, supporting a practice effect interpretation. To assess interrater reliability, intraclass correlation coefficients (ICCs) of rater agreement were calculated. This method considers raters as random effects, allowing one to generalize from a subset of raters to a universe of raters (Shrout & Fleiss, 1979). The ICCs for each RFFT part and the total scores are displayed in Table 2. Interrater reliability coefficients were excellent using Cicchetti and Sparrow s (1981) criteria for evaluating their clinical significance. Of the standard RFFT scores, the highest reliability was observed for unique designs, although the reliability coefficients for perseveration and error ratio scores were also acceptable. Excellent reliability (r ICC =.75) was also observed for the qualitative scores, with total enumerative strategies having the highest reliability using Cicchetti and Sparrow s (1981) criteria. Although still within the excellent range, the coefficients of interrater agreement were

8 886 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Table 2 Intraclass correlations coefficients (N = 90) for all indices across RFFT trails and total scores Index I II III a IV V Total Unique designs Perseverations Error ratio Rotational strategies Enumerative strategies Total clusters Mean cluster size Percent designs in strategies a Production strategy indices were not scored for RFFT Part III. slightly lower for rotational strategies scores and for mean cluster size scores. The lowest value was found for mean cluster sizes on RFFT trial IV (r ICC =.74), which is considered good. For all RFFT indices, the pattern of interrater reliability coefficients was fairly consistent across each of the RFFT parts (see Table 2). The temporal stability of RFFT scores was estimated using ICCs. This method is preferable to using Pearson-product moment correlations (PPMC) when observations are not independent (e.g., when testing the same group of persons on the same measure). In such instances, the use of PPMC can overestimate the relationship between variables (see Shrout & Fleiss, 1979). Estimates of stability were r ICC =.85 for unique designs, r ICC =.60 for perseverations, and r ICC =.46 for error ratio scores. For RFFT qualitative indices, the coefficients of stability were r ICC =.71 for rotational strategies, r ICC =.80 for enumerative strategies, and r ICC =.76 for total strategic clusters. The coefficients for percentage of designs in strategies and mean cluster size indices were r ICC =.70 and r ICC =.51, respectively. Using criteria suggested by Nunnally (1978, p. 245), the coefficients for perseverative errors and the error ratio are below the acceptable range of.70. Using the same criteria, test retest reliability coefficients for unique designs and for all production strategy scores were acceptable, with the exception of the mean cluster size score. Correlations among the RFFT scores are shown in Table 3. The perseveration and error ratio scores were highly correlated. A small negative association between the error ratio score and number of unique designs was observed. This pattern of correlations was anticipated given the interdependent manner in which these values are calculated; however, the association between unique designs and error ratio scores was quite modest. The correlation between perseverative designs and unique designs was nonsignificant (P >.05). The qualitative indices were all associated positively with unique design scores, with the total number of strategic clusters exhibiting the highest correlation (see Table 3). Production strategy scores accounted for 25% of the variance of unique designs. The use of rotational strategies was related positively to unique design output, more so than the use of enumerative strategies. Production strategy scores were unrelated to perseverations and the error ratio, with the exception of the percent designs used in strategies index, which was associated negatively with these scores (see Table 3).

9 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Table 3 Correlations among RFFT standard and production strategy scores (N = 90) Unique Persev. Error ratio Clusters Rotatn. Enumr. M Clustr. % Des. Unique Persev..06 Error ratio Clusters Rotatn Enumr M Clustr % Des Note. Unique: unique designs; Persev.: perseverative responses; Rotatn.: rotational strategies; Enumr.: enumerative strategies; Clusters: total number of strategic clusters; M Clustr.: mean strategic cluster size; % Des.: percentage of designs in strategies. P<.05. P< Discussion The present study examined the psychometric properties of the standard and qualitative scores for the RFFT in a healthy sample of college students. The means and standard deviations for RFFT scores were comparable with earlier reports (Basso et al., 1999; Ruff, 1988) and with studies that utilized samples of college students (e.g., Berning et al., 1998; Demakis, 1999). At present, there are only two previous investigations of RFFT interrater reliability for comparison. The values for RFFT unique designs, perseverative responses, and error ratios obtained in the present study were similar to those reported in Sand s (1998) unpublished dissertation, but higher than values reported by Berning et al. (1998). The raters employed in the present study differed in experience level and degree of training compared to those employed by Berning and colleagues. The raters employed by Berning et al. (1998) were college freshman that completed as few as five practice protocols as part of their training. In contrast, the expertise level and training of raters in the present study was more similar to psychometric technicians commonly employed to assist clinical psychologists with test administration (see Essig, Mittenberg, Petersen, Strauman, & Cooper, 2001; Sweet, Moberg, & Suchy, 2000). Given that the samples utilized in both investigations are very similar, the higher coefficients observed in the present study is likely attributed to rater differences. Moreover, Berning et al. (1998) reported significant increases in intraclass coefficients for raters who scores additional practice protocols, and suggested that additional training be explored as a possible means to improve the reliability of RFFT perseverative scores. The present data suggest that all RFFT indices (standard and qualitative) possess excellent interscorer agreement. Thus, both sets of indices can be scored reliably with sufficient practice in individuals with less than graduate-level training. Interrater agreement was slightly higher for enumerative strategies than that observed for rotational strategies. This finding is not surprising to those who have attempted to score production strategies for RFFT protocols. When enumerative strategies are produced, a readily identifiable set of designs appears across the page. In

10 888 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) contrast, rotational strategies require much more vigilance and perhaps spatial skills on the part of the scorer, especially on RFFT Parts IV and V where the arrangement of the five dots is not concentric. Although each drawing must have the same number of lines, the shape of the same rotated polygon (e.g., triangle) can appear quite different to the scorer (see Ruff, 1988, p.5). The interrater reliability (as well as stability) was lowest for the mean cluster size score. The correlations between this index and other scores were likely attenuated by a restriction of range due to the nature of the test stimuli. This was perhaps most evident when examining the correlation between mean cluster size and rotational strategies (r =.19, ns). Because the RFFT stimuli are composed of five dots, several types of rotational strategies may consist of three squares only, thereby constraining their cluster size to one. For example, two-line pattern rotational strategies like those shown in Figure 3b, c, and d of the RFFT professional manual have a maximum cluster size of 1 (see Ruff, 1988, p. 5). Future validity studies on this measure should assist in determining whether other test stimuli need to be constructed to better assess this proposed index. However, the mean cluster size was correlated with other RFFT indices (e.g., unique designs) to a lesser degree than was observed for the total number of clusters. Data suggest that the percentage of the total designs incorporated into strategic clusters might be a more useful index than mean cluster size in assessing the extent to which test-takers optimized strategy use. In fact, higher reliability coefficients were observed for this index. Moreover, compared to mean cluster size, the percentage of total designs included among clusters was correlated more highly with the total number of clusters and with the number of unique designs produced. The acceptable test retest reliability data for unique designs and production strategies is encouraging for practitioners and researchers who conceptualize RFFT scores as assessing relatively stable mental abilities. However, the stability of perseverations and error ratios was quite modest. This finding is true of other executive tests (e.g., Wisconsin Card Sorting Test), in which measures of perseveration have been found less reliable than indices that reflect correct responding (see Heaton, 1981; Heaton, Chelune, Talley, Kay, & Curtiss, 1993). Theorists have postulated that perseverations and other indications of poor self-monitoring and regulation are explained as artifacts of dysfunction in other cognitive domains such as working memory (e.g., Fuster, 1985; Goldman-Rakic, 1987; Pennington, Bennetto, McAleer, & Roberts; 1996) or complex attention processes (e.g., Barkley, 1996; Mirsky, 1996). The possibility that these constructs better explain perseverative tendencies could be evaluated by examining their potential role as mediator variables. Accordingly, RFFT perseverative indices could be modified to better reflect these aforementioned constructs or eliminated in favor of others. The temporal stability findings of the present study are consistent with the two previous investigations that also reported higher coefficients for unique designs as compared to perseverative responses and error ratio scores (Basso et al., 1999; Ruff, 1988). Although the pattern is similar, the coefficients of stability in the present study are slightly higher than those reported by Ruff (1988) and by Basso et al. (1999). One methodological difference that may have contributed to the higher coefficients observed in the present study is the time interval employed. The time interval between testing was considerably shorter for the present investigation (7 weeks) as compared to Basso et al. s (1999) study (12 months) and Ruff s (1988) investigation (6 months). To test the temporal stability of production strategy indices more stringently, a longer time interval should be employed.

11 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Interestingly, this is only the second psychometric study of the RFFT in the literature that addresses the short-term effects of retesting; the first is by Demakis (1999) who reported larger practice effects using a very short (3 week) interval between assessments. Frequent reassessments are often employed in post-acute rehabilitative settings (see Lynch, 1990). Moreover, forensic examinations (e.g., personal injury cases) often involve two independent assessments of the same patient within a short period of time (Putnam, Adams, & Schneider, 1992). These results do have bearing on such situations, as an average gain of eight words was observed in the retest sample. The pattern of correlations among RFFT indices was interesting and has bearing on how RFFT task demands and effective performance are conceptualized. Production strategy usage was associated positively with unique design output; however, strategy usage was not correlated with perseverative responding or with the error ratio. These findings are only in part consistent with Lezak s (1995) position that the benefits of strategy usage are two-fold: increasing the production of unique designs and decreasing the likelihood of perseverations. A negative correlation between production strategies and perseverative responses was not found, suggesting that strategy utilization does not influence perseverative responding. It is more likely that strategy usage is best conceptualized as one approach or style of responding to RFFT task demands that is frequently utilized, but not a sole means to achieve effective performance. Using strategic clusters is nonetheless a significant means to generate unique designs, as there is 25% of shared variance between these two variables. Therefore, it is likely that assessing RFFT strategy usage may be of clinical use for some persons and this possibility should be explored. Surprisingly, this was the first investigation to report bivariate correlations among RFFT indices. The nonsignificant correlation between unique designs and perseverations is consistent with Ruff s (1988) factor analytic study in which unique designs and perseverative indices clearly loaded on separate factors in a comprehensive battery of measures that assessed intelligence, attention, and executive function. Given the nonsignificant correlation between unique designs and perseverations observed in the present study, the calculation and interpretation of these scores as distinct aspects of nonverbal fluency performance appear justified. Future studies examining the factorial validity of RFFT scores could also identify the principal components best reflected by this group of indices that include redundant information. Until such components are elucidated, clinicians and researchers should bear in mind that the correlations between some of the RFFT indices are inflated due to the manner in which they are calculated. For example, the percentage of designs in clusters could be expected to be high or low depending upon whether the number of clusters is high or low. The finding that rotational strategy use was correlated more substantially with unique design output than was observed for enumerative strategy use is very interesting. Perhaps rotational strategy use requires greater spatial abilities than does enumerative strategy use, which in turn might impose more sequential or linear processing requirements? Therefore, rotational strategy use may relate more closely with nonverbal fluency overall, due to a common sensitivity to mental operations that reflect the integrity of the right anterior hemisphere. It is also possible that rotational strategies represent a more concrete solution (i.e., simply rotating the same design), while enumerative strategies are more abstract (i.e., requiring the conceptualization of new designs). Though purely speculative at present, such interpretations are plausible and deserve further study.

12 890 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) In conclusion, the psychometric properties of RFFT production strategies scores appear promising and should be investigated further. Specific issues to be explored include: examining the impact of the length of time interval on score stability, better elucidating the nature of rotational and enumerative strategies, and examining the factor structure of RFFT scores. Additionally, the low stability of RFFT perseverations and error ratio scores require further study, as does the potential role of mediating variables as described above. Finally, the clinical utility of RFFT production strategies and other validity considerations should be investigated in patient populations with known cerebral impairment. Such studies are necessary for determining the value of the RFFT qualitative indices. Acknowledgments The authors would like to acknowledge and express appreciation to Jennifer Herring, Alon Rice, Michelle Swenson, and Mona Youssef for serving as additional raters for this study. Additional thanks are expressed to Robin Montgomery and Robert Kotasek for their assistance with data collection. References Baldo, J. V., Shimamura, A. P., Delis, D. C., Kramer, J., & Kaplan, E. (2001). Verbal and design fluency in patients with frontal lobe lesions. Journal of the International Neuropsychological Society, 5, Barkley, R. A. (1996). Critical issues in research on attention. In G. R. Lyon & N. A. Krasnegor (Eds.), Attention, memory, and executive function (pp ). Baltimore, MD: Paul H. Brooks Publishing Co. Basso, M. R., Bornstein, R. A., & Lang, J. M. (1999). Practice effects on commonly used measures of executive function across 12 months. The Clinical Neuropsychologist, 13, Benton, A. L., Hamsher, K., & Sivan, A. B. (1983). Multilingual aphasia examination (3rd ed.). Iowa City, IA: AJA Associates. Berning, L. C., Weed, N. C., & Aloia, M. S. (1998). Interrater reliability of the Ruff Figural Fluency Test. Assessment, 5, Blair, J. R., & Spreen, O. (1989). Predicting premorbid IQ: A revision of the National Adult Reading Test. The Clinical Neuropsychologist, 3, Cicchetti, D. V., & Sparrow, S. S. (1981). Developing criteria for establishing the interrater reliability of specific items in a given inventory: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, Cone, J. D. (1999). Observational assessment: Measurement development and research issues. In P. C. Kendall, J. N. Butcher, & G. N. Holmbeck (Eds.), Handbook of research methods in clinical psychology (2nd ed., pp ). New York: Wiley. Demakis, G. (1999). Serial malingering on verbal and nonverbal fluency and memory measures: An analogue investigation. Archives of Clinical Neuropsychology, 14, Denckla, M. B. (1994). The measurement of executive function. In G. R. Lyon (Ed.), Frames of reference for assessing learning disabilities: New views on measurement issues (pp ). Baltimore, MD: Brooks Publishing Co. Essig, S. M., Mittenberg, W., Petersen, R. S., Strauman, S., & Cooper, J. T. (2001). Practices in forensic neuropsychology: Perspectives of neuropsychologists and trial attorneys. Archives of Clinical Neuropsychology, 16,

13 T.P. Ross et al. / Archives of Clinical Neuropsychology 18 (2003) Fama, R., Sullivan, E. V., Shear, P. K., Cahn-Wiener, D. A., Yesavage, J. A., Tinklenberg, J. R., & Pfefferbaum, A. (1999). Fluency performance in Alzheimer s disease and Parkinson s disease. The Clinical Neuropsychologist, 12, Fuster, J. M. (1985). The prefrontal cortex, mediator of cross-temporal contingencies. Human Neurobiology, 4, Goldman-Rakic, P. S. (1987). Circuitry of primate prefrontal cortex and regulation of behavior by representation memory. In F. Plum (Ed.), Handbook of physiology: The nervous system (Vol. 5, pp ). Bethesda, MD: American Physiological Society. Heaton, R. K. (1981). A manual for the Wisconsin Card Sorting Test. Odessa, FL: Psychological Assessment Resources Inc. Heaton, R. K., Chelune, G. J., Talley, J. K., Kay, G. G., & Curtiss, G. (1993). Wisconsin Card Sorting Test manual: Revised and expanded. Odessa, FL: Psychological Assessment Resources Inc. Jolly, J. K. (2000). Nonverbal creative abilities in cognition as predictors of coping response patterns in schizophrenia and in schizoaffective and bipolar disorders. Dissertation Abstracts International, 60 (11B), (University Microfilms No. AAI ). Lezak, M. D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press. Lynch, W. J. (1990). Neuropsychological assessment. In M. Rosenthal, E. R. Griffith, M. R. Bond, & J. D. Miller (Eds.), Rehabilitation of the adult and child with traumatic brain injury (2nd ed., pp ). Philadelphia: F.A. Davis Company. Mirsky, A. F. (1996). Disorders of attention: A neuropsychological perspective. In G. R. Lyon & N. A. Krasnegor (Eds.), Attention, memory, and executive function (pp ). Baltimore, MD: Paul H. Brooks Publishing Co. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill Book Company. Pennington, B. F., Bennetto, L., McAleer, O., Roberts, R. J. (1996). Executive functions/and working memory: Theoretical and measurement issues. In G. R. Lyon & N. A. Krasnegor (Eds.), Attention, memory, and executive function (pp ). Baltimore, MD: Paul H. Brooks Publishing Co. Putnam, S. H., Adams, K. M., & Schneider, A. M. (1992). One-day test retest reliability of neuropsychological tests in a personal injury case. Psychological Assessment, 4, Regard, M., Strauss, E., & Knapp, P. (1982). Children s production on verbal and nonverbal fluency tasks. Perceptual and Motor Skills, 55, Ruff, R. M. (1988). Ruff Figural Fluency Test professional manual. Odessa, FL: Psychological Assessment Resources Inc. Ruff, R. M., Allen, C. C., Farrow, C. E., Niemann, H., & Wylie, T. (1994). Differential impairment in patients with left versus right frontal lobe lesions. Archives of Clinical Neuropsychology, 9, Ruff, R. M., Light, R. H., & Evans, R. (1987). The Ruff Figural Fluency Test: A normative study with adults. Developmental Neuropsychology, 3, Sands, K. A. (1998). Nonverbal fluency: A neuropsychometric investigation. Dissertation Abstracts International, 58 (8-B), (University Microfilms No. AAM ). Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations uses in assessing rater reliability. Psychological Bulletin, 86, Spreen, O., & Strauss, E. (1998). A compendium of neuropsychological tests: Administration, norms, and commentary (2nd ed.). New York: Oxford University Press. Stuss, D. T., & Benson, D. F. (1986). The frontal lobes. New York: Raven Press. Sweet, J. J., Moberg, P. J., & Suchy, Y. (2000). Ten year follow-up survey of clinical neuropsychologists. Part I. Practices and beliefs. The Clinical Neuropsychologist, 14, Vik, P., & Ruff, R. M. (1988). Children s figural fluency performance: Development of strategy use. Developmental Neuropsychology, 4,