Practice effects and the use of alternate forms in serial neuropsychological testing

Transcription

1 Archives of Clinical Neuropsychology 20 (2005) Abstract Practice effects and the use of alternate forms in serial neuropsychological testing Leigh J. Beglinger a,, Brenda Gaydos b, Oranee Tangphao-Daniels b, Kevin Duff a, David A. Kareken c, Jane Crawford c, Philip S. Fastenau d, Eric R. Siemers b a Department of Psychiatry, University of Iowa College of Medicine, MEB, 200 Hawkins Dr., Iowa City, Iowa 52242, USA b Eli Lilly and Company, Indianapolis, USA c Department of Neurology, Indiana University School of Medicine, IN, USA d Department of Psychology, Indiana University Purdue University-Indianapolis, USA Accepted 30 December 2004 Accurate understanding of practice characteristics, performance stability, and error on neuropsychological tests is essential to both valid clinical assessment and maximization of signal detection for clinical trials of cognitive enhancing drugs. We examined practice effects in 28 healthy adults. As part of a larger study using donepezil and simulating a Phase I trial, participants were randomized into: placebo, no-treatment and donepezil. Donepezil results are presented elsewhere. Neuropsychological tests were administered in a fixed order for 6 weeks, with alternate forms available for most tests. Despite alternate forms, ANOVAs revealed significant improvements for the pooled control group (placebo and no-treatment) on all tests except Letter Number Sequencing and Trails B. Learning occurred principally in the first three to four sessions. PASAT and Stroop interference showed the greatest learning. Thus, serial assessment with alternate forms may attenuate retest effects on some tests, but continued learning occurs on novel tests or those in which an advantageous test-taking strategy can be identified. Alternate forms and baseline practice sessions may help control early, rapid improvements in clinical trials National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. Keywords: Cognition; Neuropsychological; Practice effects; Learning; Clinical trials; Phase I Corresponding author. Tel.: ; fax: address: leigh-beglinger@uiowa.edu (L.J. Beglinger) /$ see front matter 2005 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. doi: /j.acn

2 518 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Introduction Repeated neuropsychological (NP) assessments are common in clinical practice, as neuropsychologists frequently must make determinations about disease and injury progression or recovery. While assessments are typically separated by months or years, rapid interval testing is occasionally used in cases of acute injury and recovery. Additionally, serial cognitive assessments are used in the development of novel pharmaceutical treatments for conditions affecting cognition. Inherent limitations of these early Phase I investigational drug trials include the use of healthy volunteers, small sample sizes, the use of scales originally developed for patients with compromised cognition, and short retest intervals. These factors limit the study s power to detect cognitive changes in healthy volunteers, which is partially due to the learning that results from repeated exposure to the testing materials (i.e., practice effects ). Individual variability in test performance across time can also limit the findings in these Phase I studies, as intra-individual variability overcomes the drug effect. Unfortunately, there is a paucity of research in the area of serial neuropsychological assessment. Research has typically been conducted in two-session, test retest paradigms, or with experimental measures in very narrow neuropsychological domains. In these studies, age (e.g., Fozard, Vercruyssen, Reynolds, Hancock, & Quilter, 1994; Li, Aggen, Nesselroade, & Baltes, 2001) and neurological disease (e.g., Beglinger et al., 2003; Duff, Westervelt, McCaffrey, & Haase, 2001; Hultsch, MacDonald, Hunter, Levy-Bencheton, & Strauss, 2000; Strauss, MacDonald, Hunter, Moll, & Hultsch, 2002; Stuss, Stethem, Hugenholtz, Picton, & Richard, 1989) affect the stability of scores across time. Similarly, practice effects have been well demonstrated (Basso, Carona, Lowery, & Axelrod, 2002; Duff et al., 2001; Johnson, Hoch, & Johnson, 1991; McCaffrey, Ortega, Orsillo, Nelles, & Haase, 1992; Watson, Pasteur, Healy, & Hughs, 1994), and vary by age (Mitrushina & Satz, 1991), clinical group (Heaton et al., 2001), inter-test interval (Dikmen, Heaton, Grant, & Temkin, 1991; McCaffrey, Ortega, & Haase, 1993), and type of NP test (Salinsky, Storzbach, Dodrill, & Binder, 2001). Even performance in healthy volunteers is likely to be affected by regression to the mean given the moderate reliability coefficients of most cognitive tests (Salinsky et al., 2001). To maximize the utility of information obtained in early clinical studies, the typical performance of healthy volunteers must be well-characterized over time (i.e., degree of fluctuation) and the most reliable tests identified. Careful serial study of selected instruments would provide information necessary to improve the interpretation of drug versus practice effects in a clinical trial, and would help to reduce noise due to variable performance. Two study design alternatives used to attenuate practice effects are establishing a pre-drug performance plateau and the use of alternate test forms. Few studies, however, have empirically evaluated these two research designs, and those that did primarily investigated memory tests and had mixed results. For example, alternate forms have been shown to minimize practice effects on tests of declarative memory (Benedict & Zgaljardic, 1998; Crawford, Steward, & More, 1989; Fastenau, Hankins, McGinnis, Moy, & Richard, 2002; Watson et al., 1994) and spatial processing (Zgaljardic & Benedict, 2001), but not on procedural tests and verbal fluency. Practice effects generally persist on tests of nonverbal memory in healthy adults, despite alternate forms (Benedict, Schretlen, Groninger, Dobraski, & Sphritz, 1996; Benedict & Zgaljardic, 1998; Watson et al., 1994). Watson et al. (1994) found that practice

3 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) effects did not reach a plateau until 4 5 administrations and Benedict and Zgaljardic (1998) found improvement over four sessions. Claus, Mohr, and Chase (1991) also demonstrated improvement across 3 weekly test sessions in which alternate forms were used. Using a computerized battery with alternate forms, Collie, Maruff, Darby, and McStephen (2003) concluded that most improvement occurred by session 2 across cognitive domains. Thus, the number of sessions across which practice effects persist when using alternate forms is unclear. It appears that, even in healthy volunteers, the learning trajectories may differ by neuropsychological domain and age of participants. We conducted a mock Phase I study in healthy volunteers to examine: (1) learning effects and NP test performance stability on signal detection using newly created alternate forms of several common neuropsychological tests over six sessions; and (2) the time necessary to reach a learning plateau. We hypothesized that practice effects would be attenuated by the use of alternate forms and would be confined to the first two to three sessions. Healthy young adults were selected for this first study to minimize variability that is inherent with performance in older adults. 2. Methods 2.1. Participants Twenty-eight healthy adults (aged years; 71% female; 79% Caucasian) were recruited from the Indianapolis area, signed an Institutional Review Board approved informed consent to participate, and were financially compensated for their time. The average participant had some post-high school education (14.5 ± 2.1 years) and was of average intellectual endowment (Table 1), as estimated with the Wide Range Achievement Test 3rd Edition (WRAT; Wilkinson, 1993), Reading subtest (e.g. Kareken, Gur, & Saykin, 1995). All participants were screened for health status including laboratory blood work, chest X-ray, and electrocardiogram and completed a drug screen prior to enrollment in the study. Participants were excluded for significant medical or neuropsychiatric disease, use of illicit drugs or positive findings on urinary drug screening and ethanol testing, known allergies to donepezil and piperidine Table 1 Demographic characteristics of participants by treatment arm No treatment (N = 10) Placebo (N = 9) Donepezil (N = 9) P-value Age (years) 37.1 (7.9) 36.6 (7.6) 38.0 (8.0).93 a Sex (% female) 70% 78% 67% 1.0 b Education (years) 14.6 (2.7) 15.1 (1.5) 13.7 (1.7).34 a WRAT: reading 94.1 (19.9) 97.1 (12.3) 95.4 (11.3).91 a Handedness (% right) 100% 100% 89%.64 b Ethnicity (% Caucasian) 70% 78% 89%.85 b Note. Values represent means (S.D.). There were no statistically significant imbalances across treatment arms on any demographic variable at the P <.05 level. a Two-sided ANOVA. b Fisher Exact Test (two-sided).

4 520 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) derivatives, positive human immunodeficiency virus (HIV) antibodies, or current medication use that might interfere with study results. Participants had not had recent NP testing Design and procedure This investigation was part of a larger, double-blind study in which donepezil was compared to placebo and to no-treatment conditions over a period of 6 weeks. Twenty-seven subjects completed the study (8 in donepezil, 9 in placebo and 10 in no-treatment arms); one subject randomized to donepezil dropped out due to a scheduling conflict prior to Day 21. Results of the donepezil arm are presented elsewhere (Beglinger et al., 2004). Subjects were randomized across the three arms after session 3. There were no statistically significant differences across therapy assignments in demographic measures: gender, age, education, ethnic origin, and handedness. Randomization was stratified on WRAT score (standard score 90 or >90) to prevent imbalance across treatment assignments for this measure since performance on some cognitive tests is affected by level of intelligence (e.g., Horton, 1999). The NP battery was administered weekly on Days 0, 7, 14 (session 3, randomization), Days 21, 28 (session 5, end of therapy), and Day 42 (session 6, wash-out). Session 3 was chosen as the baseline because performance was hypothesized to be stable after two to three sessions of practice (which is why participants were randomized after this visit). All entered subjects were given one of two versions of the WRAT Reading subtest (blue and tan versions randomized) during the first session (Day 0) and the other at the last session (Day 42, to provide an additional stability measure and to check washout) Test materials The neuropsychological battery consisted of commonly used tests in clinical neuropsychological assessment (see Lezak, 1995; Wechsler, 1997). Tests were chosen if they were amenable to the creation of alternate forms and if they were commonly used and accepted as valid tests. Tests were administered in the following order: finger tapping (FT), verbal fluency (COWAT), Rey Auditory Verbal Learning Test (AVLT), WAIS-III Digit Symbol (Dig Sym), Trail Making Test: Parts A and B (TMT A, B), WAIS-III Letter-Number Sequencing (LNS), Stroop Color and Word Test (Stroop), and Paced Auditory Serial Additions Test (PASAT, 2.0 and 3.0 s interstimulus intervals). In addition to the original version, five alternate forms were created (6 total; Fastenau, Hankins, & McGinnis, 2001) for all tests except FT, COWAT (three forms) and Stroop in order to explore possible attenuation of practice effects with alternate forms. All alternate forms were previously validated and no form effects were found in a test retest paradigm (Fastenau et al., 2002). Form order was counterbalanced across groups, with half receiving the alternate forms in the forward sequence (e.g., Form 1 at the first session) and half in the backward sequence (e.g., Form 6 at the first session). Thus, no participant received any form more than once for tests with alternate forms. Order of test presentation within the battery was held constant. The tests were individually administered by a psychologist or research assistant, under a psychologist s supervision, in a quiet room. Day and time of test administration remained consistent for each subject throughout the study. Caffeine and tobacco use were monitored for consistency.

5 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Table 2 Raw score mean at session 1 and Pearson s correlations for sessions 1 3 by NP test for all participants (N = 28) NP test Raw score mean Session 1 vs. session 2 Session 2 vs. session3 PASAT PASAT Digit symbol Stroop color/word COWAT Trails A s Trails B s Letter Number Sequencing AVLT: Trial AVLT: Trial AVLT: long delay Finger tapping: dom. hand Note. All tests had six forms except the COWAT (three forms), Stroop (one form), and FT (one form). Mean NP performances of a randomly selected subset (N = 13 28) of this sample at session 1 were generally within half a standard deviation of the tests standardization samples (see Table 2 for session 1 raw scores), with the exception of the 3-s trial of the PASAT (0.91 standard deviations below the mean). Therefore, this sample appears representative of the healthy population. 3. Results 3.1. Placebo and no-treatment effects To determine whether a placebo effect was present on cognitive performance, a mixedeffects repeated measures model (MMRM) was used to assess placebo versus no therapy on cognition during the treatment phase (sessions 3 5) for each cognitive measure. The dependent variable was change from baseline (session 3). Model terms were treatment, visit, and treatment-by-visit interaction with the baseline score as a continuous covariate. Baseline neuropsychological functioning is believed to be an important predictor of future performance. Due to chance, it was expected that some imbalance in mean functioning across therapy arms (between groups) for some measures might be observed. Therefore, both the MMRM and ANCOVA models included the subject s baseline cognitive measure as a covariate. With this approach, the treatment difference is an estimate of what we would expect had the average baseline functioning been exactly equal across therapy arms. No statistically significant differences at the two-sided P <.05 level were observed between the no-treatment and placebo groups at session 5. Therefore, there was no evidence to support changes in performance due to a placebo effect for this study. The data across these two arms were then pooled post hoc to increase the power to explore learning effects across the six sessions. Subjects randomized

6 522 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) to the donepezil arm were not pooled with the other arms due to possible confounding effects of the drug Alternate forms To explore interform reliability for the first three sessions Pearson correlations were obtained for Forms 1, 2, and 3 and for Forms 4, 5, and 6. As described in Section 2, because of the counterbalancing of form sequence, only correlations for select forms could be made for the three visits prior to initiation of donepezil (e.g., for session 1, Forms 1 and 2 were compared for participants in the forward sequence of test administration and Forms 5 and 6 for those in the backward sequence). Most forms had adequate reliability (>.6). The strongest correlations were observed in the Dig Sym (P.93), PASAT 2-second (P.71), and Stroop: color, word, and interference (P.86). The following instruments had some forms with a correlation below 0.6: TMT A ( ), TMT B ( ), AVLT long delay ( ), and AVLT trials 1( 0.04 to 0.58) and 5 ( ), and FT ( ). It should be noted that the correlation between alternate forms was confounded with the sequence of administration for this study. Therefore, the reported estimates of reliability may not extrapolate to a study with a different administration sequence Test retest reliability Pearson correlations were first obtained for adjacent measurements between sessions 1 and 2 and sessions 2 and 3 (before donepezil was administered). These analyses are collapsed across the alternate forms in the total sample of 28 participants to explore retest reliability (Table 2). Most tests performances were highly correlated across sessions (>.65). The strongest correlations were observed for Dig Sym (P.94), PASAT 2-second (P.85), PASAT 3- second (P.84) and Stroop: color, word, and interference (P.91). The following instruments had a correlation below 0.65: TMT A, TMT B, AVLT trials 1 and 5, and long delay Practice effects Repeated measures ANOVA across six sessions (excluding subjects in the donepezil arm) revealed significant changes in at least one session, despite alternate forms, on all tests except LNS and TMT B. In addition, plots of mean scores by session for the pooled control group were visually inspected. Figure 1 shows representative plots of the raw means with 95% confidence intervals across six sessions. Due to the multiple comparisons, the more conservative alpha level of P <.01 was used (P < 0.05 was considered marginally significant). Paired t-tests (Table 3) on adjacent sessions and graphical inspection of scores indicated that learning occurred principally in the first three to four sessions. PASAT 2-second (change on adjacent sessions: P <.001, P =.005,.132,.459,.776), PASAT 3-s (change on adjacent sessions: P =.002,.114,.027,.532,.562), and the Stroop interference score (change on adjacent sessions: P =.052,.120,.005,.695,.212) showed the greatest learning. Examination of scores across sessions suggests a learning plateau for PASAT 2-s at Session 3 and for PASAT 3-s and Stroop (word reading and color/word interference) at session 4. In addition, for PASAT 3-s, a

7 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Fig. 1. Plots of mean scores on several neuropsychological instruments for the pooled control group (placebo and no treatment) for sessions 1 6. A two-week wash-out period occurred between sessions 5 and 6. Error bars represent 95% confidence intervals. Note. shorter times on Trails A and B indicate improved performance. All tests had six forms except the COWAT (three forms), Stroop (one form), and FT (one form).

8 524 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Table 3 Learning effects in healthy volunteers (pooled placebo and no treatment subjects; N = 19): raw mean change scores for sessions 1 6 NP test Session 2-1 Session 3-2 Session 4-3 Session 5-4 Session 6-5 PASAT (6.7) 3.05 (4.2) 1.32 (3.6) 0.74 (4.2) 0.16 (2.4) P <.001 P <.01 P =.13 P =.46 P =.78 PASAT (5.4) 0.79 (2.1) 1.63 (3.0) 0.32 (2.2) 0.26 (1.9) P <.01 P =.11 P =.03 P =.53 P =.56 Stroop: interference 2.74 (5.7) 2.37 (6.3) 3.47 (4.7) 0.42 (4.6) 1.68 (5.7) P =.05 P =.12 P <.01 P =.70 P =.21 AVLT: Trial (1.7) 0.05 (2.0) 1.11 (2.0) 0.58 (1.3) 0.74 (1.7) P <.05 P =.91 P =.03 P =.06 P =.08 Tapping: dom. hand 2.87 (4.8) 0.63 (3.3) 1.18 (2.9) 1.14 (2.9) 0.3 (3.3) P =.02 P =.42 P =.09 P =.11 P =.70 Digit symbol 2.79 (7.1) 1.11 (5.8) 0.42 (5.3) 4.05 (5.0) 0.63 (6.1) P =.15 P =.41 P =.73 P <.01 P =.66 COWAT 3.26 (7.8) 1.53 (7.6) 1.37 (3.9) 1.0 (7.4) 4.0 (8.9) P =.09 P =.39 P =.14 P =.56 P =.07 Trails A 2.16 (7.2) 0.53 (6.0) 1.26 (6.2) 0.63 (4.2) 0.74 (5.8) P =.21 P =.71 P =.38 P =.52 P =.59 Trails B 7.26 (24.9) 0.32 (13.8) 1.32 (16.5) 4.74 (14.3) 3.32 (16.1) P =.22 P =.92 P =.73 P =.17 P =.38 Letter Number Sequencing 0.47 (2.3) 0.32 (2.3) 0.58 (2.5) 0.21 (2.3) 0.79 (2.3) P =.38 P =.55 P =.33 P =.70 P =.15 AVLT: Trials 1 5 total 1.21 (6.1) 2.11 (6.6) 2.42 (5.6) 1.32 (6.0) 1.84 (4.9) P =.40 P =.18 P =.08 P =.36 P =.12 AVLT: long delay 0.32 (2.3) 1.21 (3.2) 0.05 (2.6) 0.58 (2.6) 0.95 (2.2) P =.56 P =.12 P =.93 P =.34 P =.08 Note. Numbers represent raw scores (S.D.) and P-values for paired comparisons t-tests. Tests are presented in order approximately from those with greatest to least practice effects. Significant and marginally significant (P <.05) change scores are bolded. Shorter times on Trails A and B indicate improved performance. PASAT, Paced Auditory Serial Additions Test; AVLT, Rey Auditory Verbal Learning Test; COWAT, Controlled Oral Word Association Test. ceiling effect likely contributed to the learning plateau, since many subjects achieved perfect performance. 4. Discussion Two important questions can be addressed through this study: (1) With alternate forms, how much improvement attributable to practice remains? and (2) How can these practice effects be attenuated or controlled? Alternate forms generally reduce practice effects compared to use of the same form, although this effect may also be moderated by test type. Benedict

9 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) and Zgaljardic (1998) suggested that alternate forms may be less effective in attenuating practice if they contain a novel concept, visuospatial learning, or graphomotor responding. They (1996) showed improvement on alternate forms of the Brief Visuospatial Memory Test- Revised across four sessions, but not on alternate forms of the Hopkins Verbal Learning Test- Revised. A comparison of three studies of visuospatial memory versus visual memory supports the conclusion that alternate forms are of less benefit for verbal memory tests (Benedict et al., 1996; Benedict & Zgaljardic, 1998; Collie et al., 2003). Fastenau et al. (2002) found practice effects on alternate forms of digit span, digit symbol, Letter Number Sequencing, TMT and the PASAT across two sessions. Consistent with those findings, we found that the PASAT and Stroop interference, tests that are most novel in procedure, were those tests with the largest practice effects. While practice effects in this study persisted longer than the two sessions reported by others, the amount and duration of improvement also varied by test. Novel tests (i.e., tasks unlikely to have been performed in everyday life by participants) and those with a large cognitive demand showed the greatest practice effects. PASAT improvement occurred over the first four sessions, albeit with the largest changes between sessions 1 and 2. Stroop interference and AVLT trial 1 also showed improvements to session 4. However, the magnitude of the changes between sessions should be noted. For the PASAT-2, average performance improved 17% between sessions 1 and 2. Improvements across the remaining sessions were approximately half the magnitude with each additional administration (8%, 3%, 2%, <1%). Similarly for PASAT-3, the difference between the first two sessions was 9%, followed by 2%, 3%, <1%, and <1%. For Stroop interference, the differences were more consistent across the first four sessions (6%, 5%, 8%) (see Table 3). Therefore, while changes persisted, the magnitude was relatively small after session 2. Interesting stability issues were also raised by examination of individual tests across the six sessions. For example, performance on the AVLT dropped over the first three sessions and the Pearson correlation between sessions 2 and 3 for AVLT trial 1 was the lowest of any measure. A non-equivalent test form is an unlikely explanation as forms were combined for that analysis and the session 1 to 2 correlation was higher. Examination of the individual participant plots across the six sessions did not reveal any outliers. However, several of the participants spontaneously commented on interference from the previous weeks lists as the study progressed. With re-testing across such a short interval, and especially with serial assessment and in healthy adults, a large pool of words in long-term memory may interfere with free recall (especially after hearing the list only once, as in the Trial 1 variable) until either a better strategy is identified or the pool exceeds the storage capacity. AVLT trial 5 and delay scores were more highly correlated across the first three sessions and may be more stable specific scores to utilize in clinical practice. Schmidt (1996) also reported the highest test retest values for the delayed recall on the AVLT (Trial 5 was lower, but higher than Trial 1). Finger tapping showed a large improvement from session 1 to 2. In this study, finger tapping was administered 3 times per hand at each session. Clinically, the standard practice is to administer many more trials, perhaps due to identification of the large practice effect. These data raise the question of whether the session 1 to 2 practice effect would have been smaller had more trials been administered at the first session or whether multiple discrete sessions are needed. This information is critical to the interpretation of motor improvements across brief intervals, as might be seen with pa-

10 526 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) tients. It also raises the question of whether tests specifically designed to be repeatable should be used over conventional tests. Tests such as the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS, Randolph, 1998) have been shown to be relatively free of practice effects (Duff, Beglinger, Schoenberg, et al., in press) over longer retest intervals (1 year). Yet Duff et al. still reported that on the Immediate and Delayed Memory Indexes of the RBANS, 11% and 9% of the sample gained 15 or more index points on retest, respectively. In the RBANS manual, Randolph noted larger practice effects for a shorter interval (38 weeks) suggesting that even repeatable batteries are not immune to practice effects on certain types of tests. While practice effects are clearly not eliminated with alternate forms, our data can be added to the accumulating literature to offer some potential strategies for minimizing the confounding effects of practice for both practitioners and researchers interested in the design of clinical trials. First, tests should be used if they are stable over time in healthy volunteers, suggesting less susceptibility to random variance. The use of alternate forms is also recommended, as many tests showed minimal improvements after session 2 when alternate forms were used. In this regard, LNS, TMT A, AVLT total learning and long delay, and COWAT were reasonably stable across the six sessions. Tests that are complicated by practice effects may also be included if they are stable after the period of initial improvement. The PASAT and finger tapping did not show significant session-to-session change after the practice effects stabilized between sessions 2 and 3. Van Gorp et al. (1993) suggested use of the dual baseline (i.e., two testing sessions administered before assessing change) in longitudinal studies to control practice effects. For these instruments, the dual baseline may be beneficial with two to three practice sessions before a treatment is measured. For clinical trials, the obvious control is to include a placebo group to enable separation of practice from drug signal changes. Second, recent research has addressed the control of practice effects with statistical methods (Basso et al., 2002; Collie et al., 2003). Several statistical approaches have been described, including the Reliable Change Index (RCI, Jacobson & Truax, 1991), Cohen s d-statistic (Benedict & Zgaljardic, 1998; Collie et al., 2003) and univariate and multivariate regression (Temkin, Heaton, Grant, & Kikmen, 1999). Temkin et al. (1999) found the RCI without correction for practice effects to be the most error-prone with the largest confidence intervals. Temkin et al. used stepwise linear regression using test retest interval, baseline performance and demographic variables produced the best prediction model. In general, older participants with lower baseline performance were poorer fits to the model. It is important to point out that most of this research has focused on test retest prediction. More work is needed to explore prediction of serial assessment. For clinical trials with more participants, it may be worthwhile to utilize the relatively complex regression models. Participants could be assessed using a dual baseline to eliminate much of the large improvements from practice; the third assessment might be used as the baseline to predict subsequent performance. This method is likely too time consuming for clinicians, who might benefit from the RCI until more research in patient samples is completed. Some limitations should be noted. While our sample was ethnically diverse, our participants were relatively young, healthy, and well educated. Previous research has shown that practice effects differ according to age and neurological, medical, and psychiatric health. Our results should be extended to include older, less educated, and lower IQ participants to determine how

11 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) these factors influence learning. We also obtained a ceiling effect on the PASAT and AVLT, a problem which could be addressed in subsequent work by using more difficult versions (e.g., faster intervals on the PASAT or superspan learning), or by employing subjects with mildly depressed neuropsychological performance, leaving room for improvement. However, regression to the mean then becomes a greater threat. The extent to which the inter-test interval may impact the amount of practice effect is also unclear and should be considered in terms of the generalizability of these results. Inter-test intervals in prior alternate forms studies have ranged from 10 min (Fastenau et al., 2001) to 1 month (Crawford et al., 1989). Cohen s d- values for a 1-day interval (0.2, Watson et al., 1994) and a 1-month interval (0.2, Crawford et al., 1989) on a verbal learning test with alternate forms in healthy volunteers were similar. Additionally, Cohen s d was similar between three visual memory tests with intervals that ranged between 1 h and 2 weeks. This supports the notion that the critical factor in the practice effect may be an interaction between experience with the test situation and the type of test, rather than mere exposure in close temporal proximity to the current assessment. Similar methodology with longer inter-test intervals is needed to ascertain whether there is a temporal effect to these patterns. Finally, while the test retest reliability of our alternate forms was acceptable, all forms were not counterbalanced for order, so we have limited data to address the true equivalence of the alternate forms. This may have been an important factor in some of the low inter-session correlations. Acknowledgements We would like to thank the Lilly Clinic CRU staff, Marianna Schneider, M.Ed., study coordinator, William Z. Potter, MD for support of these studies, Richard Mohs, PhD for his thoughtful comments, and Lu Zhang, MS and Qun Lin, MS for statistical assistance. This study was funded by Eli Lilly and Company. The study was conducted at the Lilly Laboratories for Clinical Research, Indiana University Hospital and Outpatient Center, Outpatient building, Indianapolis, IN, USA. References Basso, M., Carona, F., Lowery, N., & Axelrod, B. (2002). Practice effects on the WAIS-III across 3- and 6-month intervals. The Clinical Neuropsychologist, 16, Beglinger, L. J., Ahmed, S., Derby, M. A., Siemers, E., Fastenau, P. S., Crawford Miller, J., et al. (2003). Neuropsychological practice effects and change detection in people with schizophrenia. Schizophrenia Research, 62, Beglinger, L. J., Gaydos, B., Kareken, D. A., Tangphao-Daniels, O., Siemers, E., & Mohs, R. (2004). Neuropsychological test performance in healthy volunteers before and after Donepezil administration. Journal of Psychopharmacology, 18, Benedict, R., Schretlen, D., Groninger, L., Dobraski, M., & Sphritz, B. (1996). Revision of the Brief Visuospatial Memory Test: Studies of normal performance, reliability, and validity. Psychological Assessment, 8, Benedict, R., & Zgaljardic, D. (1998). Practice effects during repeated administrations of memory tests with and without alternate forms. Journal of Clinical & Experimental Neuropsychology, 20,

12 528 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Claus, J., Mohr, E., & Chase, T. (1991). Clinical trials in dementia: Learning effects with repeated testing. Journal of Psychiatry & Neuroscience, 16, 1 4. Collie, A., Maruff, P., Darby, D., & McStephen, M. (2003). The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test retest intervals. Journal of International Neuropsychological Society, 9, Crawford, J. R., Stewart, L. E., & Moore, W. (1989). Demonstration of savings on the AVLT and development of a parallel form. Journal of Clinical & Experimental Neuropsychology, 11, Dikmen, S., Heaton, R., Grant, I., & Temkin, N. (1991). Test retest reliability and practice effects of Expanded Halstead Reitan Neuropsychological Test Battery. Journal of International Neuropsychological Society, 5, Duff, K., Beglinger, L. J., Schoenberg, M. R., Patton, D., Mold, J., Scott, J., et al. (in press). Test retest stability and practice effects of the RBANS in a community dwelling elderly sample. Journal of Clinical & Experimental Neuropsychology. Duff, K., Westervelt, H., McCaffrey, R., & Haase, R. (2001). Practice effects, test retest stability, and dual baseline assessments with the California Verbal Learning Test in an HIV sample. Archives of Clinical Neuropsychology, 16, Fastenau, P. S., Hankins, W. T., & McGinnis, C. M. (2001). Content validity of five alternate forms for six established neuropsychological tests [Abstract]. Archives of Clinical Neuropsychology, 16(8), 824. Fastenau, P. S., Hankins, W. T., McGinnis, C. M., Moy, T., & Richard, M. (2002). Effects of alternate forms on retest effects in clinical testing [Abstract]. Journal of International Neuropsychological Society, 7(2), 151. Fozard, J., Vercruyssen, M., Reynolds, S., Hancock, P., & Quilter, R. (1994). Age differences and changes in reaction time: The Baltimore Longitudinal Study of Aging. Journal of Gerontology: Psychological Sciences, 49, Heaton, R., Temkin, N., Dikmen, S., Avitable, N., Taylor, M., Marcotte, T., et al. (2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of Clinical Neuropsychology, 16, Horton, A. (1999). Above-average intelligence and neuropsychological test score performance. International Journal of Neuroscience, 99, Hultsch, D., MacDonald, S., Hunter, M., Levy-Bencheton, J., & Strauss, E. (2000). Intraindividual variability in cognitive performance in the elderly: Comparison of adults with mild dementia, adults with arthritis, and healthy adults. Neuropsychology, 14, Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting & Clinical Psychology, 59, Johnson, B., Hoch, K., & Johnson, J. (1991). Variability in psychometric test scores: The importance of the practice effect in patient study design. Progress in Neuro Psychopharmacology & Biological Psychiatry, 15, Kareken, D. A., Gur, R. C., & Saykin, A. J. (1995). Reading on the Wide Range Achievement Test-Revised and parental education as predictors of IQ: Comparison with the Barona formula. Archives Lin Neuropsychology, 10, Lezak, M. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press. Li, S., Aggen, S., Nesselroade, J., & Baltes, P. (2001). Short-term fluctuations in elderly people s sensorimotor functioning predict text and spatial memory performance: The MacArthur Successful Aging Studies. Gerontology, 47, McCaffrey, R., Ortega, A., & Haase, R. (1993). Effects of repeated neuropsychological assessments. Archives of Clinical Neuropsychology, 8, McCaffrey, R., Ortega, A., Orsillo, S., Nelles, W., & Haase, R. (1992). Practice effects in repeated neuropsychological assessments. The Clinical Neuropsychology, 6, Mitrushina, M., & Satz, P. (1991). Effect of repeated administration of neuropsychological battery in the elderly. Journal of Clinical Psychology, 47, Randolph, C. (1998). Repeatable battery for the assessment of neuropsychological status manual. San Antonio, TX: The Psychological Corporation.

13 L.J. Beglinger et al. / Archives of Clinical Neuropsychology 20 (2005) Salinsky, M., Storzbach, D., Dodrill, C., & Binder, L. (2001). Test retest bias, reliability, and regression equations for neuropsychological measures repeated over a week period. Journal of International Neuropsychological Society, 7, Schmidt, M. (1996). Rey Auditory Verbal Learning Test: A handbook. Los Angeles: Western Psychological Services. Strauss, E., Macdonald, S., Hunter, M., Moll, A., & Hultsch, D. (2002). Intraindividual variability in cognitive performance in three groups of older adults: Cross domain links to physical status and self-perceived affect and beliefs. Journal of International Neuropsychological Society, 8, Stuss, D., Stethem, L., Hugenholtz, H., Picton, T., & Richard, M. (1989). Reaction time after head injury: Fatigue, divided and focused attention and consistency of performance. Journal of Neurology, Neurosurgery, & Psychiatry, 52, Temkin, N., Heaton, R., Grant, I., & Kikmen, S. (1999). Detecting significant change in neuropsychological test performance: A comparison of four models. Journal of International Neuropsychological Society, 5, Van Gorp, W. G., Lamb, D. G., & Schmitt, F. A. (1993). Methodological issues in neuropsychological research with HIV-spectrum disease. Archives of Clinical Neuropsychology, 8, Watson, F., Pasteur, M., Healy, D., & Hughs, E. (1994). Nine parallel versions of four memory tests: An assessment of form equivalence and the effects of practice on performance. Human Psychopharmacology: Clinical & Experimental, 9, Wechsler, D. (1997). Wechsler adult intelligence scale-iii. New York: Psychological Corporation. Wilkinson, G. S. (1993). Wide range achievement test (3rd ed.). Delaware: Wide Range Inc. Zgaljardic, D., & Benedict, R. (2001). Evaluation of practice effects in language and spatial processing test performance. Applied Neuropsychology, 8,