Negative and Positive Testing Effects in Terms of Item-Specific and Relational Information

Transcription

1 Journal of Experimental Psychology: Learning, Memory, and Cognition 2015, Vol. 41, No. 3, American Psychological Association /15/$ Negative and Positive Testing Effects in Terms of Item-Specific and Relational Information Neil W. Mulligan University of North Carolina at Chapel Hill Daniel J. Peterson Knox College Though retrieving information typically results in improved memory on a subsequent test (the testing effect). Peterson and Mulligan (2013) outlined the conditions under which retrieval practice results in poorer recall relative to restudy, a phenomenon dubbed the negative testing effect. The item-specific relational account proposes that this occurs when retrieval disrupts interitem relational encoding despite enhancing item-specific information. Four experiments examined the negative testing effect, showing the following: (a) The basic phenomenon is replicable in free recall; (b) it extends to category-cued recall; (c) it converts to a positive testing effect when the final test is recognition, a test heavily reliant on item-specific information; (d) the negative testing effect in recall, robust in a pure list design, reverses to a positive testing effect in a mixed-list design; and (_e) more generally, the present testing manipulation interacts with experimental design, such that an initially negative effect becomes positive or an initially positive effect becomes larger as the design changes from pure-list to mixed-list. The breadth of results fits well within the item-specific relational framework and provides evidence against 2 alternative accounts. Finally, this research indicates that the testing effect shares important similarities with the generation effect and other similar memory phenomena. Keywords: memory, testing effect, generation effect, recall, recognition Taking a memory test can improve subsequent memory for the material tested, a phenomenon known as the testing effect. A typical experiment on this phenomenon consists of three phases. In the first phase, participants are exposed to some material to be learned for a later test (e.g., a list of word pairs, foreign language vocabulary, educational text). In a second phase, participants are either given a test on the material or restudy it (or they may not reexperience the material at all). The third phase is a final test. Participants typically exhibit better memory for material tested during Phase 2wo compared with material restudied (or not reexperienced at all). The testing effect has been shown to be a robust phenomenon occurring for a variety of materials, various memory tests, and in real-world educational settings as well as in the laboratory (see Roediger & Butler, 2011; Roediger & Karpicke, 2006, for reviews). Although researchers have extensively explored the testing effect and especially its potential applications (e.g., to educational practice), theoretical understanding or even proper theories of the effect has lagged behind (Roediger & Butler, 2011, p. 24; cf. Halamish & Bjork, 2011; Pyc & Rawson, 2012). Relating the testing effect to other memory phenomena with more developed theoretical frameworks may help in this regard. In many experiments on the testing effect, retrieving information about a recent episode leads to better memory. Similarly, This article was published Online First September 1, Neil W. Mulligan, Department of Psychology, University of North Carolina at Chapel Hill; Daniel J. Peterson, Department of Psychology, Knox College. Correspondence concerning this article should be addressed to Neil W. Mulligan, Department of Psychology, University of North Carolina, Chapel Hill, NC nmulligan@unc.edu retrieving information from semantic memory (e.g., retrieving the antonym of hot) can enhance memory for the tested information {cold) on a subsequent recall test, the well-known generation effect (Bertsch, Pesta, Wiscott, & McDaniel, 2007; Mulligan & Lozito, 2004). Superficially, at least, the testing and generation effects appear similar: The testing effect occurs as a result of retrieving (or generating) information from episodic memory, and the generation effect occurs as a result of retrieving from semantic memory. Despite the apparent parallel, little research on the testing effect has explored this potential similarity, and those researchers who have commented on it have usually argued that the generation and testing effects are qualitatively different (e.g., Arnold & McDermott, 2013; Carrier & Pashler, 1992; Karpicke & Grimaldi, 2012; Karpicke & Zaromb, 2010; Kornell, Hays, & Bjork, 2009; Roediger & Karpicke, 2006). Peterson and Mulligan (2013), however, argued that there are important similarities between the generation and testing effects, consideration of which might shed light on the testing effect and bring to bear theories that have been successfully applied to the generation effect. In particular, Peterson and Mulligan analyzed the testing effect in terms of the item-specificrelational account, which holds the promise of embedding the testing effect in a general theory of memory encoding, providing an account of the effects of generation and several other encoding phenomena (e.g.. Hunt & McDaniel, 1993; McDaniel & Bugg, 2008; Mulligan & Lozito, 2004). The Item-Specific Relational Account Although generation typically improves memory, there are conditions under which generation fails to improve memory and even conditions under which generation produces worse recall than the 859

2 860 MULLIGAN AND PETERSON read control condition (the negative generation effect). The most successful account of the various positive, null, and negative effects of generation is based on the distinction between itemspecific and relational information (Hunt & McDaniel, 1993; Mc Daniel, Wadill, & Einstein, 1988; Mulligan & Lozito, 2004; Steffens & Erdfelder, 1998). According to this view, generation enhances encoding of item-specific features of the target item, those characteristics that differentiate the item from other items in the list and increase item distinctiveness. If the target word is generated from a cue word, generation also enhances the processing of the cue-target relation, which is particularly important in contributing to generation effects in cued recall. Interitem (or intertarget) relational processing is a second type of relational information. This refers to relationships among target items of different study trials (rather than between a cue and the target item within a study trial). Free recall of targets relies heavily on this type of relational encoding (in addition to item-specific encoding; e.g., Hunt & McDaniel, 1993; Steffens & Erdfelder, 1998). According to the item-specific relational view, the act of generation focuses encoding resources on the target item and the cue-target relation, drawing resources away from processing associations among target items (e.g., the processing of common categorical or semantic information, serial order associations, etc.).1 In delineating the types of information impacted by generation during encoding, and how these forms of information transfer differentially to various retrieval tasks, the item-specific relational account accommodates a range of positive, negative, and null generation effects. Furthermore, in addition to the generation effect, this account has been successfully applied to a number of encoding effects that produce similar patterns of results, including the perceptual-interference effect, the enactment effect, the bizarreness effect, the orthographic-distinctiveness effect, the wordfrequency effect, and the production effect (Jonker, Levene, & MacLeod, 2014; Kubik, Soderlund, Nilsson, & lonsson, 2014; McDaniel & Bugg, 2008; Mulligan & Lozito, 2004). Critically, this account delineates not only the conditions under which the generation effect occurs but also the conditions under which it reverses to produce a negative generation effect (poorer memory following generation). An example of the latter is presented by Burns (1990; Burns 1992; see also Steffens & Erdfelder, 1998). In this study, participants were presented with a series of cue-target word pairs consisting of two rhyming words (e.g., moon-spoon). In addition to the phonemic relationship between the cues and targets, there was a semantic relationship among the targets themselves: They were all exemplars from one of six different taxonomic categories. In a between-subjects manipulation, participants either read the pairs or generated the target from the rhyming cue (moon-sp ). Afterward, participants were given a free-recall test in which they were instructed to recall the target words. Compared with those who were presented with the targets intact, participants in the generation condition recalled fewer targets, a negative generation effect. According to the item-specific relational account, generation enhances item-specific and cue-target relational processing. The latter would serve to enhance encoding of the rhyming nature between the cues and targets information that is relatively unhelpful during free recall of the targets. In addition, the greater processing of item and cue-target information in the generation condition detracts from the encoding of the interitem relational information. Given the importance of this form of relational information in free recall (knowledge of the various taxonomic categories would be particularly helpful when trying to recall the targets from which they came), the result was a negative generation effect. This account is further corroborated by measures of interitem relational encoding (e.g., category clustering), which were lower in the generate than read condition. The Negative Testing Effect Peterson and Mulligan (2013) applied the item-specific relational framework to the testing effect, using it to predict the conditions under which the normally positive effects of testing would reverse, producing a negative testing effect. Peterson and Mulligan argued that a theoretical understanding of the testing effect should predict not only the presence of an effect but also the conditions under which the effect is eliminated or reversed. Given that the theoretically predicted negative testing effect was found, it appears that the item-specific relational account might be a useful addition to theoretical analyses of the testing effect. The conditions producing the negative testing effect were modeled on the conditions producing the negative generation effect. In particular, participants were presented with a series of 36 cuetarget pairs consisting of two rhyming words (e.g., moon-spoon). The target in each of pair was an exemplar from one of six taxonomic categories. In the first phase of the experiment, the word pairs were presented in a random order. In the second phase, the word pairs were presented again, blocked by the category of the target items. In the restudy condition, participants read the intact pairs again, whereas in the testing condition, participants retrieved the target given its rhyming cue (feedback was provided in case the target was not recalled). In the final phase of the experiment, participants were given a free-recall test for the targets. Because interitem relational information is an important determinant of free recall, processing the categorical structure of the targets during the second phase of the experiment (when the targets were blocked by category) would enhance final free recall. According to the item-specific relational account, participants in the retrieval condition have relatively fewer resources available for drawing associations across the word pairs. Given this, they should be less likely to notice the common categories shared by the targets, reducing the encoding of interitem relational information. This analysis generated two predictions: (a) reduced target recall in the retrieval compared with the restudy condition (i.e., a negative testing effect) and (b) less category-based clustering of the targets in the retrieval than restudy condition. Peterson and Mulligan (2013) found precisely this: The retrieval condition produced lower recall on the final free-recall test than did the restudy group a negative testing effect, and the measure of clustering on the free-recall test (assessed with the standard measure, the adjust-ratio of clustering, ARC, score) was lower in 1In typical generation manipulations, generation of the target is guided by item or cue-target information and not by interitem relational information (which is typically not a useful basis for generating targets). These typical conditions are those for which the item-specific relational account predicts disrupted interitem processing. However, in special cases interitem information may be rendered relevant for generating targets, in which case generation can actually enhance this type of information (e.g., McDaniel et al ). We return to this in the General Discussion.

3 THE TESTING EFFECT 861 the retrieval than restudy condition, consistent with reduced interitem relational encoding in the former condition. Furthermore, given that Phase 2 retrieval accuracy was not perfect, the final test data were also analyzed, conditional on correct retrieval during Phase 2. The results were unchanged: The retrieval group still recalled fewer targets on the final free-recall test, assuring that there was a negative effect on later memory not only for the general set of tested items but also for the subset of items successfully retrieved during Phase 2. Finding such a negative testing effect was novel in the literature on the testing effect2 but predicted by the item-specific relational account. Additional evidence for the item-specific relational analysis came from a second experiment in which the final test was cued recall using the original rhyme cues. Here, we propose in the present analysis that retrieval enhances cue-target processing, and thus should transfer well to a cued recall test using the original (rhyme) cues. Consistent with this prediction, a positive testing effect was found on the final test. The Present Study Although there are a few seemingly related results in the literature (see Footnote 2), the negative testing effect is novel in finding that successfully retrieving material impairs later recall of that same material. The novelty of this result and the theoretical analysis that predicted it both require additional analysis. First, the negative testing effect should be assessed for generalizability and replicability to ensure the reliability of the basic phenomenon, given the field s renewed focus on these issues (Cesario, 2014; Pashler & Wagenmakers, 2012). Second, although the negative testing effect was initially interpreted in terms of the item-specific relational account, there are at least two other viable accounts of the result. The first theoretical issue is whether the negative testing effect entails encoding differences (in interitem relational information) arising during Phase 2, as suggested by the item-specific relational account, or is strictly due to differences in category availability during the final memory test. The item-specific relational account proposes that when the organized list is presented during Phase 2, retrieving information from Phase 1 detracts from the participant s ability to notice or encode intertarget (organizational) information. In contrast, the restudy group is more likely to process this information, leading to their superior recall of the targets when the final test is free recall. In contrast to this analysis, it is possible that the effect arises during the final recall test rather than during the encoding of the organized (Phase 2) list. In particular, the restudy group may simply be more effective at using the category-level information during the free-recall test. Perhaps they are more likely to recall the names of the categories from which the targets come and use these cues to aid retrieval. That is, perhaps the restudy group is more likely to convert the free-recall test into a covert category-cued recall test. Given that category cues are known to enhance recall (e.g., Hunt & Seta, 1984; Tulving & Pearlstone, 1966), such a difference during the final test may wholly account for the advantage of the restudy group. Consequently, one important issue is to determine whether the negative testing effect can be accounted solely by retrieval differences (i.e., the availability of category cues during the final recall test) or whether one needs to propose intertarget encoding differences as well. A reanalysis of data from Peterson and Mulligan (2013), reported below, provides preliminary evidence on this issue, and Experiment 1 of the present article provides a more direct test. A second alternative account proposes that the negative testing effect may actually be a levels-of-processing effect (Craik & Tulving, 1975). This is an encoding account, but one that differs from the encoding account of the item-specific relational account. This account begins with the idea that the initial (disorganized) list presentation focuses both groups on a relatively shallow level of encoding encoding the sound-based information most apparent from the rhyming relations of the word pairs. The organized version of the Phase 2 list presentation affords greater opportunity for semantic processing if one were to focus on the category relations across target items. It may be the case that the restudy group is better situated to take advantage of this presentation than is the retrieval group. In the retrieval condition, participants are focused on retrieving information from Phase 1, which may induce greater repetition of the shallow processing engaged during the initial phase and make it less likely that these participants process the semantic (categorical) relations that exist among target items. For the restudy group, the semantic, category relations are easier to notice, so this group is more likely to process the target items in a meaningful way. In this analysis, it is not critical that the deeper encoding entails category processing merely that it represents deeper, semantic analysis than the analysis of sound or rhyme. This view states that the negative testing effect is simply a manifestation of the general levels-of-processing effect: Recall is greater when the target items are processed semantically (in the restudy condition) than nonsemantically (in the retrieval condition). This is an appealing account of the negative testing effect because it is simple, it relies on a classic explanatory concept in memory research, and it accommodates the extant data (Peterson & Mulligan, 2013). This levels-of-processing account is an encoding account, as is the item-specific relational account, but it does produce some different implications. In particular, the item-specific relational account proposes that the retrieval condition produces greater encoding of both item-specific information and cue-target relational information, with the sole disruption to in- 2 The negative testing effect reported by Peterson and Mulligan (2013) should be differentiated from other, seemingly similar, negative effects. First, this negative testing effect is not the same as the negative effects of testing demonstrated with multiple-choice tests (e.g. Roediger & Marsh, 2005). Here, participants mistakenly recall an incorrect answer as a result of selecting it on an earlier test. As with the usual testing effect, that which is retrieved (erroneously in this case) on Test 1 is more likely to be retrieved on Test 2. Second, this negative testing effect is not the same as the negative effect sometimes found following short intervals (e.g.. Roediger & Karpicke, 2006; Wheeler, Ewers, & Buonanno, 2003). Here, at short intervals and without feedback during retrieval practice, retrieval participants recall less than the restudy participants on the final recall test. This short-lasting negative effect critically relies on the absence of feedback during retrieval practice. If a participant fails to recall an item during retrieval practice, the absence of feedback dictates there is no additional opportunity to learn the forgotten item. The restudy participants, by contrast, are not burdened by this limitation; they see all the items twice and subsequently outperform their retrieval counterparts (importantly, this pattern reverses after longer delays). Studies in which the itempresentation confound is removed by either providing feedback (e.g., Carrier & Pashler, 1992) or by ensuring that initial retrieval is virtually always successful (see Karpicke, Lehman, & Aue, 2014) demonstrate a positive testing effect even at short delays. The negative testing effect of Peterson and Mulligan appears to be unique in demonstrating that retrieving targets during Phase 2 caused participants to recall fewer of the very same targets during Phase 3.

4 862 MULLIGAN AND PETERSON teritem relational information. Consequently, this analysis predicts that recognition memory for the target items, a memory test that relies heavily on item-specific information, should produce a positive testing effect. In contrast, if the negative testing effect is simply a levels-of-processing effect, we would expect to observe a negative testing effect in recognition memory because levels-ofprocessing manipulations produce the same effect on free recall and recognition (e.g., Craik & Tulving, 1975). This issue is evaluated in Experiment 2. A third goal of these studies was to assess further predictions of the item-specific relational framework. In particular, this account predicts that the negative testing effect should be moderated by experimental design, such that the negative testing effect found in a pure-list design is eliminated and perhaps becomes a positive testing effect in a mixed-list design (a prediction detailed in the introduction to Experiment 3). In Experiment 4, we assessed whether the moderation of the testing effect by experimental design applies more generally (in cases of positive rather than negative testing effects), as predicted by the item-specific relational framework. Finally, another broad goal of these experiments was to document some of the similarities between the testing and generation effects in terms of the item-specific relational framework. The present experiments demonstrate both the similarities between the testing and generation effects and the utility of the item-specific relational analysis. This in turn shows that the testing effect can be analyzed in terms of a general theoretical framework that has been applied to several other encoding effects, connecting this effect to research on other memory phenomena. Experiment 1 The negative testing effect reported by Peterson and Mulligan (2013) is important in demonstrating that information subject to retrieval practice, and indeed, items successfully retrieved, can produce worse memory on a final test of free recall. The surprising nature of that result makes it critical to assess its generality. As described below, the item-specific relational analysis predicts that the negative testing effect should also occur when the final test is a category-cued recall a test with substantial reliance on intertarget relational (in this case, categorical) processing. Furthermore, this experiment also allows us to evaluate a competing account of the negative testing effect. The item-specific relational account proposes that the negative testing effect reflects encoding differences (specifically in intertarget relational information during Phase 2). An alternative is that the two groups primarily differ during the final recall test, with the retrieval group being less likely to use category information as a retrieval cue. We address this possibility directly in Experiment 1, but additional analyses of the results of Peterson and Mulligan (2013, Experiment 1) are also relevant. In particular, when a set of to-be-recalled items consists of multiple members of each of several different categories, the number of words recalled by a given participant can be expressed as the number of categories present in the recall protocol multiplied by the average number of words recalled per category. This breakdown has been found useful in analyzing retrieval strategies and encoding effects. Furthermore, these two measures, category access and words per category, are differentially impacted by a number of variables (e.g., number of study items from each category, presence of category cues at test; Hunt & McDaniel, 1993; Hunt & Seta, 1984; Mathews & Tulving, 1973; Mulligan & Peterson, 2013; Tulving & Pearlstone, 1966). In the present case, this breakdown provides preliminary evidence against the retrieval account. In particular, if the negative testing effect is solely due to greater use of category information in the restudy condition during the final recall test, we would expect this group to exhibit greater category access during recall than the repetition condition. However, given no differences in encoding (including no differences in interitem relational encoding), the number of examples recalled per category would not be expected to differ. These expectations are based on Tulving and Pearlstone s (1966) classic demonstration that the overt introduction of category cues at test enhances category access but not words per category relative to free recall (Mathews & Tulving, 1973). Alternatively, if the negative testing effect is due to differential interitem relational encoding, then the effect should be more pronounced on words per category, a metric often affected by encoding manipulations (Hunt & Seta, 1984), than on the category access measure. The recall protocols of Peterson and Mulligan (2013, Experiment 1) were reanalyzed to yield measures of category access (the number of categories for which at least one example had been recalled) and words per category (the average number of words recalled for each category accessed). For words per category, the difference between the restudy and retrieval groups was significant (M = 3.73 and M = 3.14, respectively), t(54) = 1.90, p =.03, d =.51. For category access, this difference was not significant (M = 5.43 and M = 5.18, respectively), f(54) = 1.1, p =.14, d =.29.3 The testing manipulation influenced words per category but had relatively tittle impact on category access on the final free-recall test. This is preliminary evidence that the negative testing effect does not arise during the final free recall but rather is due to processing differences earlier (i.e., in Phase 2). Experiment 1 provides a more direct test of this issue. If the negative testing effect in free recall is solely due to the greater availability (or use) of category cues in the restudy condition, then providing all of the category cues during final recall and explicitly instructing participants to use the category cues should eliminate the effect. If, however, the difference arises during Phase 2 encoding because of differential processing of the relationship among members of common categories, then the effect should persist even with category cues. Method Participants. Seventy-six undergraduates from the University of North Carolina participated in exchange for course credit. Design and materials. Phase 2 condition (retrieval vs. restudy) was manipulated between subjects. The stimulus set consisted of 36 cue-target word pairs. The materials and procedure were modeled on Burns (1990) study of the negative generation effect. Targets (six exemplars from six different taxonomic categories) were taken from the category norms of Van Overschelde, Rawson, and Dunlosky (2004). Targets had a mean rank frequency of 4.45, and a mean frequency of 62 (Kucera & Francis, 1967). A rhyming cue was selected to accompany each target, which itself was not a member of 3 Because the relevant hypotheses are directional, both tests are onetailed.

5 THE TESTING EFFECT 863 any of the six target categories. (See Peterson & Mulligan, 2013, for a complete list of materials used). Procedure. The experiment consisted of three phases. In the first phase (which was the same for both Phase 2 conditions), participants were presented with the 36 cue-target pairs in a pseudorandomized order with the constraint that no target items in adjacent serial positions were from the same category. Participants were told the first word of each pair was the cue and the second word was the target. An example was presented showing a cue word to the left and the target word to the right (separated by a dash), to be sure that the participant knew which word was cue and which target. Each word pair was presented one at a time on a computer monitor for 4 s, with a 500-ms interstimulus interval. Participants were instructed to read the word pairs silently and remember them for a later (unspecified) memory test. After the initial presentation, participants were given a 5-min distractor task of math problems. In the second phase, participants were presented with the word pairs again in one of two manners. In the retrieval condition, participants were told they were going to practice recalling the information for the impending memory test. Here, participants were given each cue in isolation (e.g., moon-) with instructions to recall the accompanying, rhyming target presented during Phase 1. Participants read the cue aloud and recalled the target aloud as well. Participants had 10 s to retrieve the target, at which point the intact word pair (e.g., moon-spoon) was presented for 5 s. If the participant had failed to recall the target, or had inaccurately recalled the target during the allotted 10 s, she or he was instructed to read the presented (i.e., correct) target aloud. In the restudy condition, participants were told they were going to be given a second opportunity to study the words. The intact pairs were presented for 15 s each with instructions to read the word pairs aloud. Following Bums (1990), both groups were also told that the final memory test would be for the target words (i.e., the words on the right from each pair). Importantly, for both groups, the word pairs in Phase 2 were blocked by taxonomic category such that all six cue-target pairs from the silverware category were presented, followed by all six cue-target pairs from the fruit category, and the like, although the participants were not explicitly informed of this list structure. Following Phase 2, participants were given a second 15-min filler task a spatial imagery task (the clock task from Payne, Anastasi, Blackwell, & Wenger, 1994). Next, the category-cued recall test was administered. The cues consisted of the six category names as they appear in the Van Overschelde et al. (2004) category norms. Participants were told that a series of category names corresponding to target words from the study list would be presented as memory cues. Participants were asked to recall as many targets as possible for each category cue. It was made clear that each category corresponded to multiple target words. Category names were presented individually (in a new random order for each participant) for 50 s each. Participants were instructed to use the entire time to recall as many targets as possible. Results and Discussion During Phase 2, participants in the retrieval condition correctly recalled 55% of the targets. On the final recall test, the proportion of target words recalled was significantly greater in the restudy (M =.60. SE =.035) than retrieval condition (M =.51, SE =.020), t(74) = 2.24, p =.029, d =.53. That is, a negative testing effect was found on the category-cued recall test. This result persists even for a conditional analysis based on successful retrieval of the target during Phase 2, f(74) = 2.27, p =.026, d =.51. In this analysis, recall for the retrieval group is measured as the number of targets retrieved during Phase 3 that were also recalled during Phase 2, as a proportion of the number recalled during Phase 2. This conditionalized recall score (M =.49, SE =.030) was compared with the proportion of targets retrieved by the restudy group during Phase 3 (the same measure used for this group in the unconditionalized analysis). This result demonstrates that recalling a target during Phase 2 resulted in poorer memory for the same target during Phase 3. The number of intrusions was low, averaging fewer than one per participant (M =.78), and did not differ across Phase 2 conditions (tell1). The present results are important for several reasons. First, the results show that the negative testing effect reported by Peterson and Mulligan (2013) generalizes, extending to another memory test sensitive to prior interitem relational processing. Second, the conditionalized analysis also replicates the results of Peterson and Mulligan (2013), demonstrating that the negative testing effect occurs for items successfully retrieved during Phase 2 as well as for the broader set of all tested items (the unconditionalized analysis). Third, and most important, are the theoretical implications of the negative testing effect in category-cued recall. Even when all participants have explicit access to the category names and are explicitly instructed to use the category names as cues, the negative testing effect persists. This contradicts a retrieval account based solely on category availability. If the negative testing effect in free recall were caused by differential access to or use of category cues, equating access to the cues (as in category-cued recall) should eliminate the effect (and perhaps induce a more typical positive testing effect). This did not occur and implies that the negative testing effect in free recall is not a retrieval effect arising during the Phase 3 recall test. In contrast, the persistence of the effect in category-cued recall is consistent with an encoding locus of the effect, and consistent with the specific encoding account embodied by the item-specific relational analysis that the effect arises due to compromised interitem relational encoding in the retrieval condition. Experiment 2 In Experiment 2, the item-specific relational account and the account based on levels of processing is differentiated. The latter account is similar to the item-specific relational analysis in some ways. Both accounts argue that retrieval during Phase 2 biases the encoding of the organized version of the list in a way that disadvantages the retrieval group for the final recall test. The levels-ofprocessing account proposes that the demands of retrieval coupled with the rhyming nature of the word pairs induces shallow, nonsemantic encoding in the retrieval group, whereas the restudy group is more likely to engage in semantic encoding because they are less distracted with the requirements of retrieval. This account depicts the negative testing effect in both free and category-cued recall as essentially a levels-of-processing effect in which a shallow condition produces worse recall. Given that the shallow processing in the retrieval condition entails focusing on the rhyme relation between the cue and target, this account can also accom-

6 864 MULLIGAN AND PETERSON modate the positive testing effect found when the original rhyming words are used as memory cues (Peterson & Mulligan, 2013, Experiment 2). Consequently, this account is consistent with the extant results and is an appealingly simple explanation. However, the two accounts can be differentiated in at least one regard. The item-specific relational account proposes that the retrieval condition, like generation, produces greater encoding of both item-specific information and cue-target relational information, with the sole disruption to interitem relational information. Accordingly, tests that rely heavily on interitem processing (which is categorical in the present case) should exhibit the negative testing effect, as has been found in free recall and category-cued recall. Tests that rely heavily on cue-target relational information are predicted to produce a positive testing effect, as found in the rhyme-cued recall test (Peterson & Mulligan, 2013). Likewise, tests that rely heavily on item-specific information should also produce a positive testing effect. Item recognition is such a test and affords a way to contrast the two encoding accounts. Item recognition produces robust levels-of-processing effects (e.g., Craik & Tulving, 1975). If the negative testing effect is simply a levels-ofprocessing effect, we would expect to observe a negative testing effect in recognition memory. In contrast, the item-specific relational account predicts a positive testing effect on this test. The item recognition test of Experiment 2 consisted of the old target items intermixed with two types of new items: related new items from the same categories as presented in the study list and unrelated new items. The related new items were included to ensure that item recognition was not based solely on memory for the categories from the study list. Method Participants. Sixty undergraduates at the University of North Carolina participated in partial fulfillment of a course requirement. Design, materials, and procedure. Phase 2 condition (retrieval vs. restudy) was manipulated between subjects. The experiment was identical to Experiment 1 with the exception that the memory test was a recognition test. Eighteen related and 18 unrelated new items were intermixed with the 36 old target items to comprise the recognition test. The related new items consisted of three new items from each of the six categories from which the target items came. The unrelated new items consisted of three examples from each of six categories that were not previously used. Both the related and unrelated examples were common examples (all chosen from within the top 18 examples, as were the target items). Furthermore, all new items rhymed with one or more other words, so that a new item could not be excluded on the basis that it had no rhymes (and therefore could not have been part of a study list of words with rhymes). Participants were instructed to read each test word presented on the computer screen and indicate whether the word was an old word (i.e., one of the target words from the previous study phases) by hitting the y key for yes or the n key for no. The test item remained on the screen until the participant responded, initiating the next test trial. Results and Discussion During Phase 2, participants in the retrieval condition correctly recalled 59% of the targets. The results of the final recognition test are presented in Table 1. T tests indicate that the proportion of hits was significantly greater in the retrieval than restudy condition, t(58) = 2.03, p =.047, SE =.030, d =.53. Although the proportion of false alarms (FAs) was numerically lower in the retrieval than restudy condition, this difference was only marginally significant for related FAs, f(58) = 1.89, p =.067, SE =.036, d =.48, and nonsignificant for unrelated FAs, t(58) = 1.33, p =.19. The most important analyses focus on recognition accuracy, measured with the corrected hit rate (hits - FAs) separately for related FAs and unrelated FAs. Relative to either FA base rate, recognition accuracy was significantly higher in the retrieval than restudy condition, t(58) = 2.91,p =.005, SE =.044, d =.75; and, f(58) = 2.24, p =.029, SE =.050, d =.59, respectively. Comparable analyses based on d' yield identical results. In contrast to free recall and category-cued recall, the test of recognition memory produced a positive testing effect. This is important because it indicates that the negative testing effect is not simply a type of levels-of-processing effect. If the retrieval condition produces worse recall simply because it induces a shallow level of encoding, then one would expect the negative testing effect to occur on item recognition, a test that routinely exhibits the usual levels-of-processing effect. The item-specific relational account, however, argues that the retrieval condition produces greater item-specific processing and thus should lead to a positive testing effect on item recognition. This lends further support to the item-specific relational account. In addition, this result demonstrates another similarity to the generation effect. Under the encoding conditions that produce a negative generation effect in free recall, generation still produces a positive effect in item recognition (Burns, 1992; Schmidt & Cherry, 1989). Likewise, the conditions that produce a negative testing effect in free (and categorycued) recall produce a positive testing effect in recognition. Both patterns of results comport well with the item-specific relational analysis previously applied to generation (e.g., Hunt & McDaniel, 1993; Mulligan & Lozito, 2004) and currently applied to the testing effect. Experiment 3 The item-specific relational account was initially developed to account for the moderating effects of experimental design on the generation (and related) effects. In particular, when the generation manipulation is implemented in a betweensubjects (or pure-list) design, the (normally positive) effect of generation on free recall is reduced compared with when the manipulation is varied within subjects (in a mixed-list design). The reduction may take the form of an elimination of the effect or even reversal, as in the case of the negative generation effect Table 1 Experiment 2: Proportion o f Hits and False Alarms (FAs) Corrected hit rates relative to: Related Unrelated Related Unrelated Condition Hits FAs FAs FAs FAs Restudy Retrieval

7 THE TESTING EFFECT 865 (which, recall, requires a between-subjects design). This pattern has been found for the generation, perceptual-interference, enactment, production, and bizarreness effects, among others (Jonker et a! 2014; McDaniel & Bugg, 2008; Mulligan & Lozito, 2004). As applied to the negative testing effect, the item-specific relational account argues that the negative effect found in free recall with pure lists will be eliminated (and possibly converted to a more typical positive effect) for a mixed-list design. As reviewed in the introduction, the item-specific relational account argues that in pure lists, an unusual or demanding encoding condition (like the generation, enactment, perceptualinterference, testing condition, etc.) focuses processing on itemspecific (and possibly cue-target) information but disrupts processing of interitem relational processing, whereas the more usual or less demanding encoding condition (e.g., the read, observation, intact, or restudy condition) induces less itemspecific (and cue-target) processing, permitting more encoding resources to focus on interitem relational processing. In a mixed-list design, the unusual (e.g., generation) condition produces its typical trial-specific enhancements (to item-specific and cue-target processing), but the disruption caused to interitem relational information now affects not only the unusual condition but also the temporally adjacent usual (e.g., read) items. Specifically, in the present case, retrieval items disrupt the extent to which participants notice or process category information and the category relations among the target items. Consequently, the item-specific relational account attributes the negative testing effect to disrupted interitem (categorical) processing in the pure-retrieval condition compared with the pure-restudy condition. In a mixed list, the disruption of this type of encoding should extend to the interleaved restudy items. Consequently, the retrieval condition should have little relative disadvantage in interitem relational encoding, implying that the retrieval condition would no longer produce worse recall of the targets. Indeed, given that both item-specific and intertarget relational information contribute to recall, the retrieval condition might now produce greater recall (relative to the restudy condition) given its greater item-specific encoding. In Experiment 3, we manipulated retrieval or restudy in Phase 2 either between subjects (the pure-list conditions) or within subjects (the mixed-list condition), followed by a final free-recall test for the target items. The pure-list conditions constitute a replication of the original experiment in Peterson and Mulligan (2013, Experiment 1), and a negative testing effect is expected. For the mixed-list condition, the item-specific relational analysis predicts that the negative testing effect should be eliminated or rendered positive. Furthermore, two subsidiary predictions follow from the present analysis. First, the measure of interitem relational encoding (the ARC measure described below) should be greater in the pureretrieval condition than in either the mixed-list or pure-retrieval conditions. Second, given that interitem processing is predicted to be reduced in the mixed-list compared with pure-restudy condition but that item-specific processing is not expected to be affected by experimental design, recall of restudied items should be lower in the mixed-list than pure-list condition. Method Participants. Ninety undergraduates at the University of North Carolina participated in partial fulfillment of a course requirement. Design, materials, and procedure. There were three groups of participants, differentiated by their activity during Phase 2: one group engaged in restudy (pure restudy), one engaged in retrieval (pure retrieval), and a third engaged in both restudy and retrieval for different pairs (mixed list). The materials and procedures were based on those of Experiments 1 and 2. Phase 1 of the experiment was identical for all groups, and the same as in the earlier experiments. Phase 2 for the pure-restudy and pure-retrieval groups was identical to the earlier experiments. In the mixed-list group, half of the Phase 2 target pairs were presented in the restudy condition and half in the retrieval condition. This was varied randomly throughout the list with the constraint that half of the targets from each category were presented in the restudy and half in the retrieval condition, and that no more than two restudy or retrieval trials occurred consecutively. Two versions of the mixed list were constructed to counterbalance targets over the restudy and retrieval conditions. The final phase of the experiment consisted of a free-recall test in which participants were asked to recall as many of the target words as possible. Participants were reminded about the distinction between the cue and target words with reference to the example presented at the beginning of the experiment. It was made clear which of the words were to be recalled. The free-recall test lasted 5 min. Results During Phase 2, the pure-retrieval group correctly recalled 52% of the targets, and the mixed-list group correctly recalled 53% of the targets in the retrieval condition (III < 1). The results from the free-recall test are presented in Figure 1. Because Phase 2 condition is manipulated between subjects for the pure-list conditions and within subjects for the mixed-list group, the analysis requires Erlebacher s (1977) method, which produces a 2 X 2 analysis of variance (ANOVA) in which Phase 2 condition (retrieval vs. restudy) and design type (pure list vs. mixed list) are the factors. Proportion recall was submitted to this analysis, revealing a significant interaction, F(l, 86.8) = 16.05, MSE =.033, p <.001, 0, Pure List Experimental Design Mixed List Figure 1. Experiment 3: Final free-recall test. Mean proportion recalled (±SE) as a function of Phase 2 condition and experimental design.

8 866 MULLIGAN AND PETERSON T)p =.156, indicating that the testing manipulation had different effects in the mixed-list and pure-list conditions. The main effect of design was marginally significant, F(l, 86.3) = 3.38, MSE =.033, p =.07, -rip =.038, and the main effect of Phase 2 condition was not significant (F < 1). Simple tests for Phase 2 condition showed a significant negative testing effect for the pure-list groups, f(58) = 2.15, p =.035, SE =.055, d = 0.58, and a significant positive testing effect in the mixed-list condition, t(29) = 3.98, p <.001, SE =.037, d = Simple tests of design type indicated that recall in the restudy condition was significantly greater in the pure-restudy than mixed-list condition, f(58) = 4.01, p <.001, SE =.055, d = 1.07; the trend for recall in the retrieval condition to be lower in the mixed-list than purerestudy condition was not significant, f(58) = The recall data were also conditionalized on correct recall during Phase 2 (which affects only the retrieval conditions), producing M =.35 (SE =.033) and M =.37 (SE =.045) for the pure-retrieval and mixedlist retrieval conditions, respectively. This analysis produced the same pattern of results as the unconditionalized data. The number of cue words inadvertently recalled (M = 0.68) and the number of other intrusions (M = 0.72) were analyzed with one-way ANOVAs using group (pure restudy, pure retrieval, mixed) as a between-subjects factor. Neither type of error significantly differed across groups (ps >.10). Interitem relational processing is reflected in the amount of category clustering found in recall protocols, typically measured with the ARC (Hunt & McDaniel, 1993). The ARC score has a value of 0 for chance-level clustering, positive values for abovechance clustering, and a value of 1 for perfect clustering. ARC scores were submitted to a one-way ANOVA using group as a between-subjects factor, revealing a significant effect, F(2, 83) = 5.41, MSE =.243, p =.006, rjp =.115. Post hoc tests (Fisher s least significant difference [LSD]) indicated significantly higher ARC scores in the pure-restudy condition (M =.58) than in either the pure-retrieval (M =.16) or mixed-list (M =.21) condition, the latter two conditions not differing from one another. Additionally, ARC scores were significantly greater than chance in the purerestudy condition, f(28) = 5.08, p <.001, SE =.11, d =.95, and in the mixed-list condition, r(27) = 2.52, p =.018, SE =.08, d =.48, but were only marginally greater than chance in the pureretrieval condition, f(28) = 1.92, p =.065, SE =.08, d =.36.4 Discussion The pure-list conditions were identical to those used in Peterson and Mulligan (2013, Experiment 1) and replicate those results in detail: (a) The proportion recalled demonstrated a negative testing effect, (b) the ARC scores were lower in the pure-retrieval than pure-restudy condition, and (c) the negative testing effect was still found when final recall in the pure-retrieval group was conditionalized on successful retrieval in Phase 2. These findings join the results of Peterson and Mulligan (2013) and the present Experiment 1 in showing that for tests with a heavy reliance on interitem relational encoding, the negative testing effect occurs. More importantly, the negative testing effect was moderated by experimental design as predicted by the item-specific relational account. In particular, this account proposes that under mixed-list conditions, the disruption to interitem relational encoding induced by retrieval will affect temporally adjacent restudy trials as well. This should reduce differential interitem encoding, the driving force of the negative testing effect, placing the two encoding conditions on a more equal footing in this regard, thus allowing the superior item-specific encoding of the retrieval condition to manifest as a positive testing effect. The recall results match this prediction: A significant negative testing effect in the pure-list conditions turned into a significant positive testing effect in the mixed-list group. If the retrieval items in the mixed list disrupt interitem relational encoding, this implies that the mixed-list condition should exhibit less category clustering than the pure-study condition, a result borne out in the ARC score analysis. Finally, the predicted decrement in recall for restudied items in the mixed-list compared with pure-list condition was also found. Experiment 4 In Experiment 3, we tested the prediction that experimental design moderates the negative testing effect. However, this prediction applies not only to the negative version of the testing effect, but rather is a general prediction. If the effect in the pure-list condition is a negative effect (e.g., the negative testing effect), this account predicts that the effect becomes less negative, perhaps positive, in the mixed-list condition. If the effect is null or positive to begin with (i.e., in the pure-list conditions), the account predicts that the effect becomes larger (more positive) in the mixed-list condition. To evaluate this more general prediction, we need to start with a pure-list condition that does not produce a negative testing effect. Consequently, the materials used in the prior experiments were presented in a modified way that prior research shows eliminates the negative testing effect in a pure-list design. In particular, during Phase 2, the pairs were presented in a random rather than blocked order. Elimination of the blocked order eliminates the negative testing effect on the final free-recall test (Peterson & Mulligan, 2013, Experiment 3). This occurs because participants in the pure-restudy condition no longer have an advantage in noticing and encoding the intertarget categorical information. Thus, the restudy group loses a potent advantage for free recall of the targets, and the negative testing effect is eliminated. In Experiment 4, we manipulated retrieval or restudy in Phase 2 either between subjects (the pure-list conditions) or within subjects (the mixed-list condition). On the basis of prior research, we expect a null or positive testing effect in the pure-list condition. The item-specific relational analysis predicts that the (positive) testing effect will be larger in the mixed-list than pure-list condition. Furthermore, because the list is presented in a random order in both phases, categorical information is much less likely to play a role in retrieval (Peterson & Mulligan, 2013, Experiment 3). Rather, participants are likely to use other, less effective forms of interitem relational information during retrieval such as order information that has been shown to contribute to free recall when categorical information is not salient (McDaniel & Bugg, 2008; 4 ARC scores could not be computed for four participants (two from the mixed-list condition and one each from the pure-list groups) because these participants either recalled no more than one example from each category accessed, recalled words from only a single category, or overtly organized their answers on the test sheet. In these cases, the ARC score is not computable (see Roenker et al 1971).

9 THE TESTING EFFECT 867 Nairne, Reigler, & Serra, 1991). This information is typically less effective for recall with longer lists, but its effects are still observable (Mulligan & Lozito, 2007). A secondary goal of Experiment 4, therefore, was to assess the use of order information in free recall. The item-specific relational analysis argues that this form of interitem relational encoding should likewise be affected by the testing manipulation, such that the pure-restudy condition should produce greater order encoding than either the mixed-list condition or pure-retrieval condition. This would provide a converging assessment of the prediction that testing disrupts interitem relational processing now with order rather than categorical information (see Karpicke & Zaromb, 2010, for related evidence). Method Participants. Eighty-four undergraduates at the University of North Carolina participated in partial fulfillment of a course requirement. Design, materials, and procedure. There were three groups of participants, differentiated by their activity during Phase 2: one group engaged in restudy (pure restudy), one engaged in retrieval (pure retrieval), and a third engaged in both restudy and retrieval for different pairs (mixed list). The methods were identical to Experiment 3 with the following modifications. In the second phase of the experiment, the word pairs were not blocked by taxonomic category of the targets. Word pairs were arranged in a pseudorandom order such that no two consecutive trials contained targets from the same taxonomic category (this was a different random order than in Phase 1). Otherwise, Phase 2 was identical to Experiment 3. Results and Discussion During Phase 2, the pure-retrieval group correctly recalled 49% of the targets, and the mixed-list group correctly recalled 48% of the targets in the retrieval condition (Id < 1). The proportion of target words recalled on the final test (see Figure 2) was analyzed via Erlebacher s (1977) method with a 2 (Phase 2 condition) X 2 (design type) ANOVA. The analysis revealed a significant main effect of Phase 2 condition, F(l, 80.2) = 29.67, MSE =.019, p <.001, -rip =.27, indicating greater recall for the retrieval than restudy condition that is, a positive testing effect was found. The interaction was also significant, F(l,80.2) = 8.67, MSE =.019, p =.004, -rip =.10, indicating that the testing effect was greater in the mixed-list than pure-list condition (the main effect of design was not significant; F < 1). Simple tests for Phase 2 condition showed a significant (positive) testing effect in the mixed-list condition, /(27) = 6.82, p <.001, SE =.032, d = 1.29, and only a nonsignificant trend for a positive testing effect in the pure-list condition, r(54) = 1.59, p =.12. Simple tests of design type indicated that recall in the retrieval condition was significantly greater for the mixed-list than pure-retrieval condition, f(54) = 2.20, p =.032, SE =.025, d =.59; there was a trend for recall in the restudy condition to be lower in the mixed-list than purerestudy condition, r(54) = 1.75, p =.087, SE =.030, d =.47. The recall data were also conditionalized on correct recall during Phase 2 (which affects only the retrieval conditions), producing M =.40 {SE =.030) and M =.50 {SE =.031) for the pure-retrieval and mixed-list retrieval conditions, respectively. This analysis produced the same pattern of results as the unconditionalized data. Neither the number of cue words inadvertently recalled (M = 1.18) nor the number of other intrusions (M = 0.61) significantly differed across groups (ps >.25). ARC scores did not differ among the groups (M = -.01), p >.25, nor did mean ARC scores differ from zero within any of the groups {ps >.20), indicating little reliance on interitem categorical information during recall. The influence of order information in free recall is typically assessed with the input-output (IO) correspondence score (De- Losh & McDaniel, 1996; Nairne et al., 1991). The IO score assesses the extent to which recall output corresponds to serial input position, with a chance-level value of scores were submitted to a one-way ANOVA using group as a betweensubjects factor, revealing a significant effect, F(2, 82) = 6.44, MSE =.017, p =.003, Tip =.14. Post hoc tests (Fisher s LSD) indicated significantly higher IO scores in the pure-restudy condition (M =.59) than in either the pure-retrieval {M =.47) or mixed-list {M =.49) condition, the latter two conditions not differing from one another. Additionally, IO scores were significantly greater than chance in the pure-restudy condition, t{27) = 3.77, p =.001, SE =.023, d =.71, but did not differ from chance in the other two conditions {ps >.20). Several aspects of the present results merit comment: First, considering just the pure-list conditions, the presentation of List 2 in a random order eliminated the negative testing effect, producing instead a trend toward a positive testing effect, and reduced ARC scores to chance, indicating little encoding and use of categorical interitem information. These results are similar to those reported in Peterson and Mulligan (2013, Experiment 3) and indicate that eliminating the restudy advantage in categorical processing, a potent form of interitem encoding for free recall of targets, eliminates the negative testing effect.5 Pure List Mixed List Experimental Design Figure 2. Experiment 4: Final free-recall test. Mean proportion recalled (±SE) as a function of Phase 2 condition and experimental design. 5 The present analysis assumes that categorical processing is a more potent basis for free recall in Experiment 3 than was order information (another form of interitem relational processing) in the present experiment. Correlational analyses are consistent with this assumption. In Experiment 3, the pure-list groups exhibited strong, significant correlations between ARC scores and recall of r(27) =.52 and of r{21) =.55 for the restudy and retrieval groups, respectively. In Experiment 4, the correlations between IO scores (the only significant form of relational processing) and recall scores merely trended (nonsignificantly) positive, r{26) =.15 and of r(25) =.30 for the pure-restudy and pure-retrieval groups, respectively.

10 868 MULLIGAN AND PETERSON Second, and more central for present purposes, the results are consistent with the predictions of the item-specific relational account about the moderating effects of experimental design. As predicted, the testing effect was greater for mixed lists than pure lists. This result complements that of Experiment 3 in which the pure-list conditions produced a negative testing effect, and the change to a mixed-list design rendered a positive testing effect. It was possible that this pattern was a limited consequence of the highly unusual starting conditions in which pure lists produce a negative testing effect. Experiment 4 indicates that the moderating effect of experimental design is more general and occurs outside the realm of the negative testing effect. The 10 scores also provide converging evidence supporting the item-specific relational account. Order information is another form of interitem information, the encoding of which should be disrupted by retrieval in Phase 2. Consistent with this implication is the finding that the 10 scores were significantly greater for the pure-restudy than pure-retrieval or mixed-list conditions. Finally, the present results reveal another similarity between the testing and generation effects: The testing effect in recall was larger in a mixed-list than a pure-list design, and 10 scores in recall were greater in the pure-restudy condition compared with the pure-retrieval condition (and the mixed-list condition). Both results are consistent with research on the generation effect and several other effects (e.g., the enactment, perceptual-interference, and production effects) encompassed by the item-specific relational account (e.g., Engelkamp & Dehn, 2000; Jonker et al., 2014; McDaniel & Bugg, 2008; Mulligan, 1999; Nairne et al., 1991). General Discussion The negative testing effect reported by Peterson and Mulligan (2013) is an important reversal of the typical positive effects of testing, predicted by a general theory of memory encoding effects. The present study had several goals: first, to replicate and assess the generality of the negative testing effect; second, to evaluate additional predictions of the item-specific relational account; third, to contrast these predictions with two alternative accounts; and fourth, to assess some potential similarities between the testing and generation effects. Experiment 1 revealed that the negative testing effect extends to category cued recall, whereas Experiment 2 demonstrated the more typical positive effect of testing for recognition memory. Experiment 3 showed that the negative testing effect is moderated by experimental design: Pure lists produced a negative testing effect for final free recall, whereas mixed lists produced a positive testing effect. Experiment 4 demonstrated that when interitem categorical information is less salient, the negative testing effect is eliminated for pure lists (Peterson & Mulligan, 2013, Experiment 3). More importantly, the (now positive) testing effect was larger for mixed lists than pure lists. With regard to the first goal, the present study provides a direct replication of the negative testing effect in free recall for the pure-list conditions of Experiment 3. In addition, Experiment 1 shows that the effect generalizes to the category-cued recall test. But there are limits to this generalizability. The negative testing effect does not hold when the Phase 2 list is randomized (Experiment 4), when testing is manipulated in a mixed list (Experiment 3), or for memory tests such as recognition memory (Experiment 2) or cued recall with the original rhyme cues (Peterson & Mulligan, 2013, Experiment 2). Of course, all of these results stem from predictions of the item-specific relational analysis, as detailed next. The negative, null, and positive testing effects found in Peterson and Mulligan (2013) and in the present study follow from the item-specific relational account. This account proposes that the testing condition focuses processing on the target item and the within-pair relationship, impairing the processing of interitem relational information. However, this account argues that retrieval engages greater processing of cue-target information as well as itemspecific information for the target words. This account maps on well to the results of the present experiments and those of Peterson and Mulligan (2013). When the Phase 2 lists were organized, such that testing was expected to disrupt interitem, categorical processing, the testing group produced lower free recall (Peterson & Mulligan, Experiment 1; the present Experiment 3). This reduced free recall was accompanied by reduced ARC scores, the standard metric of listwide organizational processing. Likewise, these results are consistent with the negative testing effect found with category-cued recall, a test similarly dependent on intertarget (categorical) processing. When category information is no longer salient (when the Phase 2 list is presented in a random order), the restudy group loses a potent advantage, and the negative testing effect is eliminated (Peterson & Mulligan, Experiment 3; the present Experiment 4). But even under these conditions, another form of interitem processing, order information (as assessed by the IO score), is disrupted by testing (either in a pure list or intermixed with restudy trials) relative to a pure-restudy list. This converges with the results of Karpicke and Zaromb (2010), who found that compared with restudy, testing produced worse performance on another measure of order encoding, the order reconstruction test. For tests having minimal reliance on interitem relational encoding but instead relying on either cue-target relational or itemspecific information, this analysis predicts a positive testing effect. Cued recall using the original cue words, which relies on cuetarget relational encoding, reveals a positive testing effect (Peterson & Mulligan, 2013, Experiment 2). Likewise, recognition memory, relying heavily on item-specific information and minimally on interitem relational information, also exhibited a positive testing effect (Experiment 2). Another prediction of the item-specific relational account is that the negative testing effect should be moderated by experimental design. When retrieval and restudy trials are intermixed during Phase 2, the disruption to interitem relational encoding caused by retrieval trials should interfere with the encoding of this information for the adjacent restudy items as well. This general reduction in interitem relational encoding for all items in the mixed list robs the restudy items of an advantage for later free recall, allowing the superior item-specific encoding of retrieved items to manifest in a positive rather than a negative testing effect. The results of Experiment 3 concurred: Pure lists produce a negative testing effect, whereas the mixed list produced a positive testing effect. Furthermore, the pure-restudy condition produced higher ARC scores than the other two groups, consistent with reduced interitem relational encoding in both the pure-retrieval and mixed-list conditions. Of course, the prediction that design moderates the testing effect is a general prediction of the item-specific relational account, not a special prediction regarding the negative testing effect. Experiment 4 showed more generally that design moderates the testing

11 THE TESTING EFFECT 869 effect even when the pure-list condition does not begin with a negative testing effect. The results of Experiment 4 are also consistent with the results of Karpicke and Zaromb (2010), who in separate experiments implemented a pure-list and a mixed-list testing manipulation. The testing effect was significant in both designs but was nearly twice as large in the mixed-list than pure-list experiment. This issue was not the focus of the Karpicke and Zaromb (2010) study, so no statistical analysis was performed across experiments. Consequently, the present Experiments 3 and 4 are the first direct tests of this issue. The use of the rhyme pairs in the present experiments also provides a check on the generality of this pattern of results. The apparent moderation of the testing effect by experimental design seen in Karpicke and Zaromb s (2010) results was produced with pairs that were semantically related (e.g., knee-bone, although targets across pairs were unrelated), whereas the pairs used in the present study were phonologically related, indicating that this result (moderation by design) generalizes across different associative relationships. With this in mind, we should reconsider some earlier discussions of the generation effect in the testing literature. In particular, it has been proposed that the generation and testing effects are qualitatively different because generation often produces a null effect on free recall in pure-list designs, whereas testing, even in pure-list designs, typically produces a significant effect in recall (Karpicke & Zaromb, 2010; Roediger & Karpicke, 2006). This may indeed be an important difference, but focusing too heavily on it might obscure the similarity uncovered in the present experiments, specifically that both the testing and generation effects are moderated by experimental design as predicted by the item-specific relational account. The present results are not only consistent with the item-specific relational analysis but also argue against two alternative accounts of the negative testing effect: a retrieval-based explanation and an account based on levels of processing. The first of these proposes that the negative testing effect arises during the final free-recall test because the restudy condition makes greater use of (covert) category cues to yield an advantage over the testing group. Preliminary evidence against this view comes from the reanalysis of the free-recall data reported in the introduction to Experiment 1: Category access was not significantly affected by testing, whereas words per category were. If the negative testing effect was due to differential use of category cues at Phase 3 retrieval rather than to differential encoding during Phase 2, we would expect greater category access in the restudy condition coupled with equivalent retrieval of words per category (e.g., Hunt & Seta, 1984; Tulving & Pearlstone, 1966). More direct evidence comes from the category-cued recall results, which indicate that when all category cues are explicitly provided, the negative testing effect persists. The retrieval view predicts that equal access to the category cues should eliminate the negative testing effect on the final test. Its persistence indicates that the degree of categoricalrelational encoding must have differed across the groups. It should be noted that this rejected account is a retrieval-only account of the negative testing in free recall. That is, we are not generally rejecting a role for retrieval across the range of results reported here. Indeed, traditional notions of transfer-appropriate processing and the importance of encoding-retrieval match are embedded in the item-specific relational framework. Different types of encoding (e.g., enhanced interitem relational processing) are hypothesized to transfer particularly well to some forms of retrieval (e.g., free recall) but less well to others (e.g., recognition memory). Consequently, the item-specific relational account proposes that the testing manipulation used here has various effects on the encoding of different forms of information, which in interaction with retrieval circumstances give rise to the various results documented. In this very general sense, the item-specific relational account depicts the negative testing effect (as well as the other effects accommodated by this theoretical framework) as akin to classical effects of transfer-appropriate processing (Morris, Bransford, & Franks, 1977) or encoding specificity (Tulving & Thomson, 1973). The second alternative account that we have rejected depicts the negative testing effect as a levels-of-processing effect. Under this view, the testing group engages in shallower processing than the restudy group. The demands of retrieval during Phase 2 biases this group against processing the categorical organization available in the second list presentation. In contrast, the restudy group is not under the demands of retrieval and thus is more likely to process this semantic aspect of the words. This analysis accommodates the negative testing effect in free and category-cued recall. However, this account also predicts a negative testing effect in item recognition, given the pervasive finding of levels-of-processing effects in this test. The presence of a positive testing effect in item recognition argues against a simple levels-of-processing account. A final goal was to explore some potential similarities between the testing and generation effects as part of the evaluation of the testing effect in terms of the item-specific relational account. First, of course, is the fact that both the testing effect and the generation effect reverse to negative effects under the same conditions (Bums, 1990, 1992; Peterson & Mulligan, 2013; Steffens & Erdfelder, 1998; the present Experiment 3). Second, the conditions rendering a negative generation or testing effect in free recall render a positive effect in recognition memory (Burns, 1992; Schmidt & Cherry, 1989; the present Experiment 2). Third, both the testing effect and the generation effect can be moderated by experimental design, such that the effect is smaller (or reversed) in pure-list design and larger in a mixed-list design (the present Experiments 3 and 4). Finally, under similar conditions, both the testing and generation manipulations disrupt measures of interitem relational encoding, such as the ARC and IO scores and order reconstruction (Karpicke & Zaromb, 2010; Naime et al., 1991; Peterson & Mulligan, 2013; the present Experiments 3 and 4). Despite some differences between generation and testing effects (cf. Karpicke & Zaromb, 2010), there are important similarities in the effects that bear further exploration under unifying theoretical frameworks. Speaking of potential difference between the testing and generation effects, or perhaps more accurately between different instantiations of the testing manipulation, some studies indicate that testing can enhance organizational processing, a result at odds with the present findings. In particular, when categorized lists are repeatedly recalled (using free recall) or restudied, the testing condition produces greater category clustering (as assessed with the ARC score) on the final recall test (Zaromb & Roediger, 2010; see also Congleton & Rajaram, 2012). This discrepancy is presumably due to the difference in the implementation of the testing manipulation, using free recall in these studies and cued recall in the present experiments (and in Karpicke & Zaromb, 2010). Free

12 870 MULLIGAN AND PETERSON recall of an entire set of targets focuses processing on the interrelations between the targets, whereas cued recall of individual targets typically does not. This implies that some testing conditions enhance interitem relational processing and others disrupt such processing. Does this, then, indicate a potential difference between retrieving from episodic memory (the testing effect) and semantic memory (the generation effect)? Such a conclusion is premature and raises two issues. First, as noted in Footnote 1, the item-specific relational account is predicated on typical generation manipulations in which interitem relational information is either not a useful basis for generation or, if potentially useful, is obscured by more dominant generation cues (as in the case of the manipulation producing the negative generation effect). When interitem relational information is rendered relevant for the generation task, the account predicts that it will be used and thus enhanced in the generation condition, a result initially documented by McDaniel et al. (1988). Consequently, in certain implementations, generation actually enhances rather than disrupts interitem relational information. Second, and more generally, when the testing and generation effects are implemented in similar ways (e.g., through cued recall of targets), the effects demonstrate a number of similarities, including the disruption of interitem relational processing (and others reviewed earlier). When testing is implemented via free recall, then interitem relational processing can be enhanced. To determine whether this is a point of difference between the testing and generation effects, the generation manipulation would have to be implemented in a similar, unconstrained manner. Methodologically, this may not be possible, as typical generation manipulations carefully constrain semantic retrieval to ensure that the read and generate conditions process the same items (i.e., to avoid item selection confounds). To implement retrieval from semantic memory for a set of targets in an unconstrained manner (ala free recall) might make it impossible to tame item selection problems. However, it should be noted that this is a practical problem, not a principled issue. In principle, one would need to compare unconstrained semantic retrieval with unconstrained episodic retrieval of a set of items to determine whether the aforementioned issue really represents a difference between the generation and testing effects. Conclusions The testing effect has been extensively studied, but theoretical analysis has lagged behind (Roediger & Butler, 2011). The present article, motivated by potential similarities between the testing and generation effects, applied the item-specific relational account to delineate some conditions under which negative and positive testing effects might be found. The success of these predictions indicates that it may be fruitful to consider the testing effect in terms of this general account, which has been applied successfully to a number of seemingly disparate memory effects (Hunt & McDaniel, 1993; Jonker et al., 2014; McDaniel & Bugg, 2008; Mulligan & Lozito, 2004). Furthermore, the present results point to similarities between the subsequent mnemonic effects of episodic retrieval (the testing effect) and semantic retrieval (the generation effect). References Arnold, K. M., & McDermott, K. B. (2013). Test-potentiated learning: Distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, doi: /a Bertsch. S., Pesta, B. J., Wiscott, R.. & McDaniel, M. A. (2007). The generation effect: A meta-analytic review. Memory & Cognition, 35, doi: /bf Burns, D. J. (1990). The generation effect: A test between single- and multifactor theories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, doi: / Burns, D. J. (1992 ). The consequences of generation. Journal of Memory and Language, 31, doi: / x(92)90031-r Carrier. M., & Pashler, H. (1992). The influence of retrieval on retention. Memory & Cognition, 20, doi: /bf Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, doi: / Congleton, A., & Rajaram, S. (2012). The origin of the interaction between learning method and delay in the testing effect: The roles of processing and conceptual retrieval organization. Memory & Cognition, 40, doi: /s y Craik, F. 1. M., & Tulving, E. (1975). Depth of processing and retention of words in episodic memory. Journal of Experimental Psychology: General, 104, doi: / DeLosh, E. L & McDaniel. M. A. (1996). The role of order information in free recall: Application to the word-frequency effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, doi: / Engelkamp, J., & Dehn, D. A. (2000). Item and order information in subject-performed tasks and experimenter-performed tasks. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26, doi: / Erlebacher. A. (1977). Design and analysis of experiments contrasting the within- and between-subjects manipulation of the independent variable. Psychological Bulletin, 84, doi: / Halamish, V & Bjork, R. A. (2011). When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, doi: /a Hunt, R. R & McDaniel. M. A. (1993). The enigma of organization and distinctiveness. Journal of Memory and Language, 32, doi: /jmla Hunt, R. R & Seta, C. E. (1984). Category size effects in recall: The roles of relational and individual item information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, doi: / Jonker, T. R., Levene, M.. & MacLeod, C. M. (2014). Testing the itemorder account of design effects using the production effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, doi: /a Karpicke, J. D., & Grimaldi, P. J. (2012). Retrieval-based learning: A perspective for enhancing meaningful learning. Educational Psychology Review, 24, doi: /sl Karpicke, J. D., Lehman, M & Aue, W. R. (2014). Retrieval-based learning: An episodic context account. In B. H. Ross (Ed.), The psychology of learning and motivation (Vol. 61, pp ). San Diego, CA: Elsevier Academic Press. Karpicke, J. D.. & Zaromb, F. M. (2010). Retrieval mode distinguishes the testing effect from the generation effect. Journal of Memory and Language, 62, doi: /j.jml Komell, N., Hays, M. J., & Bjork, R. A. (2009). Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, doi: / aoo15729 Kubik, V., Soderlund, H., Nilsson, L.-G., & Jonsson, F. U. (2014, February 7). Individual and combined effects of enactment and testing on memory