Empirical Analysis of the Intrinsic-Extrinsic Distinction of Judgments of Learning (JOLs): Effects of Relatedness and Serial Position on JOLs

Transcription

1 Journal of Experimental Psychology: Learning, emory, and Cognition 2001, Vol. 27, No. 5, Copyright 2001 by the American Psychological Association, Inc /01/S5.00 DOI: 10.1O37// Empirical Analysis of the Intrinsic-Extrinsic Distinction of Judgments of Learning (JOLs): Effects of Relatedness and Serial Position on JOLs John Dunlosky and Greg atvey University of North Carolina at Greensboro A. Koriat's (1997) cue-utilization framework provided a significant advance in understanding how people make judgments of learning (JOLs). A major distinction is made between intrinsic and extrinsic cues. JOLs are predicted to be sensitive to intrinsic cues (e.g., item relatedness) and less sensitive to extrinsic cues (e.g., serial position) because JOLs are comparative across items in a list. The authors evaluated predictions by having people make JOLs after studying either related (poker-flush) or unrelated (dog-spoon) items. Although some outcomes confirmed these predictions, others could not be readily explained by the framework. Namely, relatedness influenced JOLs even when manipulated between participants, primacy effects were evident on JOLs, and the order in which blocks of items were presented (either all related items first or all unrelated items first) influenced JOLs. The authors discuss the framework in relation to these and other outcomes. etacognitive monitoring plays a central role in the control of thought and action, including aspects of problem solving, learning and memory, and text comprehension (Hacker, Dunlosky, & Graesser, 1998; azzoni & Nelson, 1998; etcalfe & Shimamura, 1994). Accordingly, interest has been increasing in metacognitive judgments that tap people's monitoring of different aspects of cognition and in the functional role of these judgments in selfregulation (Nelson & Narens, 1990). The present research focuses on Koriat's (1997) cue-utilization framework for understanding judgments of learning (JOLs), which are metacognitive judgments that measure a person's monitoring of his or her own memory (for functional accounts of JOLs, see Son & etcalfe, 2000; Thiede & Dunlosky, 1999). ore specifically, JOLs are an individual's predictions about the likelihood of subsequent memory performance for recently studied items. Uncovering the psychological bases of JOLs is an important goal of metacognitive research, partly because these judgments are related to people's control of learning, which may in turn influence performance in many settings. Although the seminal research on JOLs was conducted over 30 years ago (Arbuckle & Cuddy, 1969), advances toward a general theory of JOLs have been meager, with the bulk of theoretical developments being focused on explaining individual phenomena. In a recent article, Koriat (1997) provided one of the most significant advances in theory about how people make JOLs, moving well beyond previous research by developing a framework that John Dunlosky and Greg atvey, Department of Psychology, University of North Carolina at Greensboro. any thanks go to R. Hunt, S. cdonald, and the participants of the UNCG Cognitive Research Group for stimulating conversations about this research. We also thank A. Benjamin and W. Kelemen for helpful comments on a previous version of this article. Correspondence concerning this article should be addressed to John Dunlosky, P.O. Box 26164, Department of Psychology, University of North Carolina, Greensboro, North Carolina Electronic mail may be sent to dunlosky@uncg.edu. unifies seemingly unrelated phenomena. Koriat's cue-utilization framework can account for various dissociations between JOLs and memory performance (Benjamin, Bjork, & Schwartz, 1998), the delayed-jol effect (Nelson & Dunlosky, 1991), the predictive validity of one person's JOLs to another's performance (Lovelace, 1984), the relationship between generation fluency and JOLs (atvey, Dunlosky, & Guttentag, 2001), and increases in JOL accuracy that occur across study-test trials (King, Zechmeister, & Shaughnessy, 1980). The Cue-Utilization Framework and the Intrinsic-Extrinsic Distinction The cue-utilization framework accounts for the effects of various factors on JOLs, such as relatedness or the number of study presentations, in the following manner. When a person makes a JOL, he or she presumably uses various cues to infer the likelihood of subsequent memory performance (Schwartz, Benjamin, & Bjork, 1997). A core assumption is that the kind of cue critically influences JOLs. An intrinsic cue involves "the characteristics of the study items that are perceived to disclose the items' a priori ease or difficulty of learning" (Koriat, 1997, p. 350). An extrinsic cue involves "factors that pertain either (a) to the conditions of learning, or (b) to the encoding operations applied by the learner" (Koriat, 1997, p. 350). Intrinsic cues and extrinsic cues are expected to differentially influence JOLs. In particular, Koriat (1997) proposed that JOLs are comparative in nature: They tend to focus on the relative recallability of different items within a list and are less sensitive to factors that affect overall performance... Therefore, in general, the effects of extrinsic factors should be discounted in predictions of recall relative to those of intrinsic factors, (p. 352) Two ideas in the quote above appear most relevant to the intrinsic-extrinsic distinction. First, intrinsic cues and extrinsic cues influence JOLs through a comparison process that occurs across items. An example serves to illustrate the plausibility of this 1180

2 THEORY OF JUDGENTS OF LEARNING 1181 assumption. Consider an individual studying dog-spoon and responding with a JOL of 50%. If subsequently the individual studies poker-flush, he or she may compare the relatedness across items (e.g., this related item will be more memorable than the previous unrelated item) and hence respond with a higher JOL of 80%. Without experiencing the levels of a cue (in this case, unrelated vs. related word pairs) within a list, JOLs would presumably not be adjusted on the basis of an analysis of the cue, partly because the individuals would not have access from memory about the levels of the cue and hence would not be expected to accurately judge how the manipulation affects memory (Ericsson & Simon, 1980). Second, this comparison process is implicated as producing the differential effects of intrinsic versus extrinsic cues on JOLs. Namely, the comparison process is more sensitive to factors that affect the differential recall of individual items (i.e., intrinsic cues) than to factors that affect overall recall (i.e., extrinsic cues). Koriat (1997) did not provide a detailed account of how the comparison process results in differential sensitivity of the two kinds of cue or empirical support for the necessity of this comparison process. Accordingly, in the present research, we empirically evaluated whether a comparison process is required to observe the effects of a factor on people's JOLs. The intrinsic-extrinsic distinction accounts for several new findings reported by Koriat (1997). In one experiment, participants studied paired-associate items and immediately after studying each item made a JOL. Two within-subject factors were manipulated: (a) Items were either easier to learn or difficult to learn, and (b) items were presented on two study-test trials. JOLs were expected to be sensitive to the effect of item difficulty (an intrinsic cue) on recall, whereas JOLs were expected to be relatively insensitive to the effect of number of presentations (an extrinsic cue) on recall. Results confirmed these expectations. Recall was 29% greater for items rated as easier to learn than for items rated as more difficult to learn, and JOLs were 35% greater for items rated as easier to learn than for items rated as more difficult to learn. By contrast, although recall was 30% greater after two presentations than after one presentation, JOLs were only 16% greater after two presentations than after one. Thus, as compared with the effect of these factors on recall, JOLs were more sensitive to item difficulty than to the number of presentations. These outcomes confirmed predictions from the cue-utilization framework; however, no other empirical evaluation of the framework has been reported. Our major goal was to explore the bases of people's JOLs by testing a priori predictions from the cue-utilization framework. In three experiments, we evaluated predictions that pertain to the distinction between intrinsic and extrinsic cues. Each participant studied paired associates (e.g., dog-spoon) and made a JOL immediately after studying each, which included presenting the stimulus (i.e., dog-?) of an item and asking the participant to predict the percentage likelihood of correctly recalling the response on an upcoming test. We report the degree to which the magnitude of JOLs was influenced by (a) item relatedness using both betweensubjects and within-subject comparisons, (b) the order in which items were presented during study (related items first vs. unrelated items first), and (c) the serial position of items during study. We now discuss the significance of these factors in evaluating the cue-utilization framework. As highlighted below, keys to this endeavor included classifying each factor as an intrinsic or extrinsic cue and deciding whether participants can compare levels of each cue across to-be-judged items. Predictions Concerning the Focal Factors: Judgments of Learning and Discounting The strategy used in the present research was to examine both the effect of a given factor on JOLs and whether JOLs discounted the effect of that factor on memory performance. Although demonstrations of the effects of various factors on JOLs have been abundant in the literature, the notion of discounting was recently introduced by Koriat (1997) to evaluate the cue-utilization framework and hence requires some explanation. Discounting is related to the absolute accuracy of JOLs in that the influence of a factor on the magnitude of JOLs is compared with the influence of that factor on memory performance. JOLs discount the effects of a factor whenever they underestimate the magnitude of the factor's influence on memory performance, regardless of the actual levels of JOLs and memory performance. If the factor has a smaller influence on JOLs than on memory performance, discounting has occurred. By contrast, if the influence of the factor on JOLs and memory performance is identical, then JOLs did not discount the effects of that factor on memory performance. This conclusion would hold even if the magnitude of the JOLs underestimated memory performance at each level of the relevant factor, which would indicate less-than-perfect absolute accuracy. We now describe each factor and corresponding predictions from the cueutilization framework. Item Relatedness and Experimental Design Item relatedness refers to the degree of semantic relationship between the two words of a paired associate (henceforth referred to as relatedness). For instance, the pair table-chair would be considered related, whereas the pair dog-spoon would be considered unrelated. Relatedness has already played a prominent role in investigations of JOLs (Carroll, Nelson, & Kirwan, 1997; Connor, Dunlosky, & Hertzog, 1997; Hertzog, Kidder, Powell-oman, & Dunlosky, in press; Koriat, 1997; Rabinowitz, Ackerman, Craik, & Hinchley, 1982). The consistent outcome of these investigations is that JOLs are substantially greater for related than unrelated items. The effect of relatedness on recall performance also does not appear to be discounted by JOLs (e.g., Connor et al., 1997; Koriat, 1997). Because it seems intuitively reasonable to classify relatedness as an intrinsic cue (although we verify this assumption using a technique used by Koriat, 1997), these outcomes are consistent with the cue-utilization framework. In the present research, we attempted to replicate them. A core assumption of the cue-utilization framework that underlies the differential effects of intrinsic versus extrinsic cues is that JOLs are comparative in nature. Thus, relatedness is not expected to influence JOLs when it is manipulated between subjects because JOLs "focus on the relative recallibility of different items within a list" (Koriat, 1997, p. 352). This prediction is also suggested by the process of criterion shift (cf. Carroll & Nelson, 1993). To evaluate it, we blocked the items during study so that some participants studied related items first, whereas other participants studied unrelated items first. Thus, the effects of relatedness on JOLs could be examined between subjects by comparing JOLs for related

3 1182 DUNLOSKY AND ATVEY items from participants who studied related items first with JOLs for unrelated items from those who studied unrelated items first. This design also provides another factor that is relevant to evaluating the cue-utilization framework. In particular, the order in which the blocks of items are presented is a between-subjects factor: Some participants received related items first (followed by unrelated items), and other participants received the opposite order. The order of the blocks is an extrinsic cue and hence is expected to have a relatively minimal influence on JOLs. Serial Position of Paired-Associate Items During Study Serial-position effects have greatly informed theories of memory. In stark contrast, serial-position effects on JOLs have not yet been examined. One reason for this omission is that research on JOLs has almost exclusively involved paired-associate learning, which may be less likely to elicit classical serial-position effects on recall performance. Recent evidence by Brooks (1999), however, has demonstrated primacy effects for paired-associate recall both for related items and for unrelated items (see also Brooks, 1994). Will a primacy effect also occur for JOLs? According to the intrinsic-extrinsic distinction, serial position is an extrinsic cue. That is, the position of an item on a list pertains "to the conditions of learning" (Koriat, 1997, p. 350) and hence is expected to have a relatively minimal effect on JOLs. oreover, whereas relatedness was not expected to be discounted by JOLs, primacy effects on recall were expected to be discounted. In contrast to this prediction, previous research on metacomprehension judgments suggests that JOLs will be influenced by serial position. aki and Berry (1984) had students read a chapter from a text, which was divided into multiple sections. After reading a section, each student rated how well he or she would perform on a test over the material in that section. The correlations between serial position of the section and the ratings were reliably less than zero. oreover, serial position and test performance were not related, indicating that the judgments did not discount this extrinsic cue. Of course, the bases of JOLs and metacomprehension judgments are apparently not identical (e.g., Kelemen, Frost, & Weaver, 2000), so the degree to which a primacy effect will occur for JOLs is still an open and critical question. Experiment 1 Besides examining the influence of the aforementioned factors on JOLs, we first established that our manipulation of relatedness was an intrinsic cue. Although on modest reflection one might classify relatedness as an intrinsic cue, objectively verifying such an intuition is possible within the cue-utilization framework. Because a factor is an intrinsic cue if it entails a priori differences in rated difficulty of learning across levels of that factor, participants' judgments about how easy each item is to learn (which are made prior to study) can be used to verify that relatedness is an intrinsic cue (as in Koriat, 1997). If it is, these ease-of-leaming (EOL) judgments will be greater for related than for unrelated items. Thus, we had participants make an EOL judgment for each item prior to the critical study-test trial. By collecting EOL judgments and JOLs, we were also able to extend research conducted by Leonesio and Nelson (1990). They had participants make EOL judgments for 20 unrelated noun-noun pairs, study the pairs during multiple study-test trials (half to a criterion of one correct recall; the other half to a criterion of four correct recalls), and then make JOLs for each pair. The mean correlation between EOL judgments and JOLs was.19, suggesting "these metamemory judgments are based on underlying structures that are multidimensional" (Leonesio & Nelson, 1990, p. 468). One dimension that may be shared by both kinds of judgments is an individual's beliefs about normative item difficulty, which may be activated by manipulating the relatedness of items. If so, substantial correlations will be found between EOL judgments and JOLs when the judgments are correlated across all items. By contrast, the correlations are expected to be diminished when computed for a given kind of item (as in Leonesio & Nelson, 1990). Although investigating relations between judgments was exploratory, outcomes here were also relevant to classifying factors as intrinsic cues. We leave further consideration of this topic for the General Discussion. Participants and Design ethod The design of Experiment 1 was a 2 (relatedness: related items or unrelated items) X 2 (order: related items first or unrelated items first) mixed-model factorial with relatedness as a within-subject factor and order as a between-subjects factor. Sixty undergraduates from the University of North Carolina at Greensboro (UNCG) participated to fulfill a course requirement. Thirty participants were randomly assigned to the two order groups by order of appearance. Because of a record-keeping error, 31 participants were assigned to the group receiving related items first, and 29 participants were assigned to the group receiving unrelated items first. aterials and Apparatus Lists consisted of 60 paired associates. The 30 related items consisted of nouns that shared a semantic association (e.g., pig-cow); the 30 unrelated items consisted of nouns that shared no obvious semantic association (e.g., roof-pope). A list was constructed for each participant. Each list was blocked by kind of item (related or unrelated). Each block contained 30 items, with the order of items being randomized for each participant. acintosh computers presented instructions and items and collected all responses. Procedure Participants were tested individually and were instructed that they would be studying a list of pairs for an upcoming test of paired-associate recall. Participants completed the tasks in the order that they are described below. Ease-of-learning judgments. Participants first made an EOL judgment for each item. An EOL judgment was prompted by the entire item (e.g., roof-pope) near the top of the screen, along with the query, "How difficult do you believe the word pair above will be to learn so that you can successfully remember the second word when prompted with:." The cue for the pair then followed (e.g., for roof pope, roof was shown), and participants were instructed to make the judgment on a scale from 0 (very difficult to learn) to 100 (very easy to learn). Participants could use any whole number from 0 to 100. Study. The order of the items within each block was randomized anew for study. Participants were instructed to study each pair for an upcoming paired-associate recall test. Each item was presented individually for 5 s. After the offset of the presentation of an item for study, a JOL was made. Judgments of learning. A JOL was prompted by an entire item presented near the top of the screen, along with the query, "How confident are

4 THEORY OF JUDGENTS OF LEARNING 1183 you that in about ten minutes you will be able to remember the second word of the pair above when prompted with:." The stimulus cue for the item followed, and participants were instructed to make the judgment on a scale from 0 (definitely will not recall) to 100 (definitely will recall). Participants could use any whole number from 0 to 100. Paired-associate recall. Following the study-jol phase, participants performed an unrelated filler task for 5 min, which was followed by paired-associate recall. The order of items was randomized anew, and for each item, the cue was presented (e.g., roof-), and the participant was instructed to respond with the word that was originally paired with that cue (i.e., pope). Participants were given unlimited time to respond to each cue. Recall Performance Results Percentage of correct recall performance was computed for each participant. eans across participants' values are reported in Table 1. A 2 (relatedness) X 2 (order) analysis of variance (ANOVA) revealed a main effect of relatedness, F(l, 58) = 164.8, = 0.02, whereas the main effect of order and the interaction were not reliable, Fs < 1.0, s < To examine serial-position effects, we collapsed across each set of three serial positions to reduce error (as in Brooks, 1999), which yielded 10 serial-position bins. For each participant, the mean percentage of correct recall was calculated for items within each bin. eans across participants' values are presented in Figure 1. Given that the order of presentation of the cues during the test was randomized, we did not expect recency effects, and hence our inferential analyses involved primacy effects alone. In particular, we expected declines in performance across the first two to three serial-position bins for both kinds of item (Brooks, 1999), so our analyses included recall performance from only the first three serial-position bins. Because Brooks (1999) found primacy effects and because they involve a decrease in recall across initial serial positions, one-tailed analyses were planned. However, unless otherwise noted, all primacy effects declared as reliable had p <.05, using a standard two-tailed comparison. In contrast to expectations, inspection of Figure 1 suggests that primacy effects did not occur for both kinds of item but instead were isolated to unrelated items. A 2 (relatedness) X 3 (position) X 2 (order) ANOVA revealed a main effect of relatedness, F(l, 58) = 81.0, = 0.10; however, all other effects and interactions were not reliable, Fs < 2.6, including the Position X Relatedness interaction, F(2, 116) = 1.7, = 0.05, p =.19. Thus, primacy effects were not reliably observed in the omnibus ANOVA for either kind of item. Although these outcomes are apparently inconsistent with those reported by Brooks (1999), the expected trends were evident for unrelated items. The presentation of items for EOL judgments may have boosted recall and hence diminished primacy effects for related items. In particular, the median level of recall of related items was 1.0 for five (out of six) of the first three serial-position bins, suggesting ceiling effects may have obscured any primacy effect for these items. agnitude of the etacognitive Judgments Although our main focus was on JOLs, we first present results concerning EOL judgments to establish relatedness as an intrinsic cue. Next, analyses of JOLs are reported, which are relevant to evaluating the cue-utilization framework. Ease-of-learning judgments. For each participant, we computed the median EOL judgment separately for related items and for unrelated items. When related items were presented first, means across participants' values were 75 for related items and 19 for unrelated items. When unrelated items were presented first, mean values were 85 for related items and 43 for unrelated items (all four s < 4). A 2 (relatedness) X 2 (order) ANOVA revealed main effects for relatedness, F(l, 58) = 258.0, = 285.5, and for order, F(l, 58) = 20.2, = 419.3, and a reliable interaction, F(l, 58) = 5.3, = ost important, the interaction did not qualify the main effect of relatedness, with EOL judgments being greater for related than unrelated items regardless Table 1 Recall Performance and the agnitude of the Judgments of Learning Recall performance Judgment of learning Related Unrelated Related Unrelated Item order Experiment 1 Related/unrelated Unrelated/related Experiment 2 Related/unrelated Unrelated/related Experiment 3 ixed list Note. eans are percentages across participants. Item orderrefersto which block of items (either related items or unrelated items) was presented first during study. For Experiment 3, the order of items was not blocked but instead was mixed throughout the presentation of items. Related = related items; unrelated unrelated items.

5 1184 DUNLOSKY AND ATVEY Related items presented first Related items Recall - -O - - JOLs a Unrelated items Unrelated items presented first f a A i- ^ ^ / / * '-" ^ Serial-Position Bins Serial-Position Bins Figure 1. For Experiment 1, serial-position curves are presented for recall and for judgments of learning (JOLs) as a function of kind of item (either related pair or unrelated pair) and the order in which the blocks of items (either related first or unrelated first) were presented for study. of order, t& > These outcomes empirically establish relatedness as an intrinsic cue (as in Koriat, 1997). Judgments of learning. To evaluate the effects of relatedness and presentation order on the magnitude of JOLs, we computed the median across each participant's JOLs. eans across participants' values are reported in Table 1. A 2 (relatedness) X 2 (order) ANOVA revealed that relatedness reliably influenced JOLs, F(l, 58) = 240.6, = 285.8, which replicates previous research and is consistent with predictions from the cue-utilization framework. The main effect of order was also reliable, F(l, 58) = 17.3, = 543.3, whereas the Relatedness X Order interaction was not reliable, F(l, 58) = 2.2, = Concerning the former, JOL magnitudes were greater when unrelated items were presented first than when related items were presented first. Why does the order of presentation influence people's JOLs? We offer an answer to this question in the General Discussion. Finally, consider the between-subject comparison of relatedness. JOLs were reliably greater for related items presented first ( = 69) than for unrelated items presented first ( = 38), r(58) = 5.5, which is inconsistent with a core assumption of the cue-utilization framework. We analyzed the serial-position data for JOLs as in the analyses of recall performance. The corresponding curves for JOLs are presented in Figure 1. Primacy effects on JOLs were more apparent for unrelated items than for related items. A 2 X 3 X 2 ANOVA revealed a main effect of group, F(l, 56) = 11.9, = , and a main effect of relatedness, F(l, 56) = 168.3, = 873.3, which correspond to the main effects on JOL magnitude described above. ost important, the main effect for position was reliable, given the planned, one-tailed analysis, F(2, 112) = 3.0, = 194.0, p =.055, and the Position X Relatedness interaction was reliable, F(2, 112) = 12.8, = Follow-up ANOVAs conducted separately for related and unrelated items indicated that the effect of position was not reliable for related items, F(2, 116) = 1.4, = 200.9, but was reliable for unrelated items, F(2, 112) = 15.2, = The primacy effects on JOLs for unrelated items suggest a caveat to interpreting the critical between-subjects effect of relatedness on JOL magnitude. Perhaps this effect is solely due to averaging across serial positions. ore specifically, when averaging across serial positions in computing JOL magnitude, the overall differences in relatedness may arise solely from declines in JOLs across serial positions for the unrelated items. If so, JOL magnitude for the first serial position may not differ in the between-subjects comparison of related versus unrelated items. In contrast to this possibility, the JOL magnitude for items in the first serial position was reliably greater for related items ( = 71; = 3.9) when they were presented first than for unrelated items ( = 54; = 5.5) when they were presented first, f(48) = 2.5.

6 THEORY OF JUDGENTS OF LEARNING 1185 Relationship Between Ease-of-Learning Judgments and Judgments of Learning To examine the relation between EOL judgments and JOLs, we computed a gamma correlation between the two judgments for each participant. Three gamma correlations were computed: one for related items, one for unrelated items, and one across all items. eans across participants' correlations are reported in Table 2. Three outcomes are noteworthy. First, the relation between the two kinds of judgment was consistently greater than 0, suggesting that the judgments share a common basis. Second, the values computed within a given kind of item were reliably greater for related items than for unrelated items, F(l, 51) = 5.58, = The main effect of order and the Relatedness X Order interaction were not reliable, Fs < Third, and most important, when the correlations were computed across all items, the magnitude of the correlations were relatively substantial, both in comparison with values computed for a given kind of item and in comparison with the value of.19, reported by Leonesio and Nelson (1990). Thus, the composition of the list of to-be-studied items apparently moderates the relation between these judgments, with the relation being higher when the list consists of both related and unrelated items than when the list consists of only one subset of items. In the General Discussion, we discuss the significance of these outcomes to theories of JOLs. Relative Accuracy of the Judgments of Learning Although relative accuracy of the JOLs was least relevant to accomplishing the particular goals of our research, we briefly describe it to connect with the extant literature. Relative accuracy was operationalized as a Goodman-Kruskal gamma correlation between a participant's JOLs and his or her own recall performance. Gammas were computed separately for related items, for unrelated items, and across all items. eans across participants' gammas are reported in Table 3. A 2 (relatedness) X 2 (order) ANOVA revealed a main effect of relatedness, F(l, 44) = 5.8, = 0.19, indicating gammas were greater for unrelated than related items. This outcome was also recently demonstrated by Hertzog et al. (in press), who explained the effects of relatedness on relative accuracy in terms of people's reliance on extrinsic cues. In particular, people may base JOLs on whether a mediator (e.g., an image or sentence) was generated Table 2 ean Gamma Correlations Between Ease-of-Learning Judgments and Judgments of Learning in Experiment 1 Item order Related/unrelated Unrelated/related Al.33 Related items Unrelated items All items Note. eans are across intraindividual gamma correlations between easeof-learning judgments and judgments of learning. Item order refers to which block of items (either related items or unrelated items) was presented first during study. Related = related items; unrelated = unrelated items. Table 3 Relative Accuracy: ean Gamma Correlations Between Judgments of Learning and Recall Performance Item order Related/unrelated Unrelated/related Related/unrelated Unrelated/related ixed list Related items Experiment Experiment Experiment 3.06 Unrelated items All items Note. eans are across intraindividual gamma correlations between judgments of learning and recall performance. Item order refers to which block of items (either related items or unrelated items) was presented first during study. For Experiment 3, the order of items was not blocked but instead was mixed throughout the presentation of items. Related = related items; unrelated = unrelated items. while studying a paired associate, with JOLs being greater for items that elicited a mediator than for items that did not elicit a mediator (cf. Dunlosky, Kubat, & Hertzog, 2000). Because mediator production is apparently more diagnostic of subsequent recall performance for unrelated than related items (Dunlosky & Hertzog, 1998), the use of this cue as a basis for JOLs will yield higher accuracy for unrelated items. Although exploration of this effect will likely prove informative to theories of JOLs, because relative accuracy is not central to our present goals, we do not discuss these outcomes further in the present research. The main effect of order and the interaction were not reliable (Fs < 1.8, s < 0.22). Also, it should be noted that relative accuracy was greater across all items than when it was computed individually for either kind of item. This latter outcome is not surprising, given that the magnitude of people's JOLs was greater for related than unrelated items (Table 1). Discussion Several outcomes from Experiment 1 are especially relevant for theories of JOLs. In terms of the cue-utilization framework, EOL judgments discriminated between related and unrelated items, demonstrating that relatedness is an intrinsic cue (as in Koriat, 1997). With this in mind, however, the relatively large relationship between EOL judgments and JOLs across items suggests difficulties in using the former in classifying a factor as an intrinsic cue a point we expand upon in the General Discussion. Furthermore, relatedness had a substantial effect on JOLs, which is consistent with a prediction from the cue-utilization framework. Other outcomes, however, appeared inconsistent with this framework. Between-subjects comparisons demonstrated substantial effects of relatedness on JOLs. For unrelated items in which ceiling effects were not evident, primacy effects were either as large (or even larger) for JOLs than for recall, indicating that this extrinsic.03

7 1186 DUNLOSKY AND ATVEY cue is not necessarily discounted. Both outcomes are inconsistent with predictions from the cue-utilization framework. oreover, order of presenting items (an extrinsic cue) influenced JOLs and not recall, an outcome that cannot be explained by any current theory of JOLs. Experiment 2 Collecting EOL judgments was valuable for establishing relatedness as an intrinsic cue (Koriat, 1997) as well as for further describing the relationship between EOL judgments and JOLs (Leonesio & Nelson, 1990). However, the extra presentation of items required for collecting EOL judgments may have had an undue influence on recall for related items, which in turn limited the interpretation of some of our outcomes. oreover, any reactive effects of making EOL judgments may also misrepresent the influence of key factors on JOLs. Thus, we evaluated the predictions again by using the identical design as in Experiment 1, with the exception that EOL judgments were not collected. Participants and Design ethod Sixty undergraduates from UNCG participated to fulfill a course requirement. The design was a 2 (relatedness) x 2 (order) mixed-model factorial with item as a within-subject factor and order as a between-subjects factor. Thirty participants were assigned to each group by order of appearance. aterials and Procedure Items from Experiment 1 consisted of a list of 60 paired associates. Lists were blocked by item (related or unrelated) with the order of items within a block randomized for each participant. The procedure of Experiment 2 was identical to that used in Experiment 1, except that EOL judgments were not collected in Experiment 2. Recall Performance Results Percentage of correct recall was computed for each participant, and means across participants' values are reported in Table 1. A 2X2 ANOVA revealed a main effect of relatedness, F(l, 58) = 331.9, = 0.01, whereas the effect of order, F(l, 58) = 1.1, = 0.05, and the interaction, F(l, 58) = 3.6, = 0.01,p =.06, were not reliable. Analyses of serial-position effects were conducted as described in Experiment 1. eans across participants' values are presented in Figure 2. In contrast to Experiment 1, primacy effects for recall performance were evident in all conditions. Reliable main effects occurred for relatedness, F(l, 58) = 173.8, = 0.10, and for 100 Related items presented first Related items Unrelated items Recall --O-- #_. Unrelated items presented first Serial-Position Bins Serial-Position Bins Figure 2. For Experiment 2, serial-position curves are presented for recall and for judgments of learning (JOLs) as a function of kind of item (either related pair or unrelated pair) and the order in which the blocks of items (either related first or unrelated first) were presented for study.

8 THEORY OF JUDGENTS OF LEARNING 1187 position, F(2, 116) = 6.1, = The Relatedness X Order interaction was also reliable, F(l, 58) = 4.8, = ore important, all interactions involving position and the main effect of order were not reliable, Fs < Thus, although a primacy effect is not evident across the first three serial-position bins when unrelated items were presented first, primacy effects overall were reliable and replicate outcomes reported by Brooks (1999). Judgments of Learning We evaluated the effects of relatedness and presentation order on the magnitude of JOLs as in Experiment 1. eans across participants' values are reported in Table 1. Relatedness reliably influenced JOLs, F(l, 58) = 134.5, = The main effect of order was reliable, F(l, 58) = 5.3, = 999.4, and the Relatedness X Order interaction approached reliability, F(l, 58) = 3.9, = 315.7, p =.052. Finally, consider the betweensubject comparison of relatedness. The magnitude of JOLs was reliably greater for related items when presented first ( = 61) than for unrelated items when presented first (Af = 36), r(58) = 3.6. The serial-position curves for JOLs are presented in Figure 2. A 2 X 3 X 2 ANOVA revealed a main effect of order, F(l, 55) = 8.5, = , a main effect of relatedness, F(l, 55) = 115.9, = 903.0, and a reliable Order X Relatedness interaction, F(l, 55) = 13.4, = ost important, the main effect of position was reliable, F(2, 110) = 7.9, = 337.2, indicating the presence of reliable primacy effects on JOLs. Although the primacy effect appeared more pronounced for unrelated items when they were presented first (bottom of right-hand panel) than for any of the other conditions, no interactions involving serial position were reliable, Fs(2, 110) < 2.1, s < Relative Accuracy of the Judgments of Learning Gamma correlations were computed separately for related items, for unrelated items, and across all items. eans across participants' gammas are reported in Table 3. A 2 (relatedness) X 2 (order) ANOVA was conducted. The main effect of relatedness approached reliability, F(l, 57) = 3.1, = 0.15, p =.08, with a trend toward higher values for unrelated than related items. The main effect of order and the interaction were not reliable, Fs < 1.0, s < Relative accuracy was also greater when computed across all items than when it was computed individually for either subset of item, which was expected, given that relatedness influenced both JOL magnitude and recall performance. Discussion In Experiment 1, ceiling effects were apparent for recall of related items across individual serial positions, which may have concealed primacy effects. In contrast, recall performance was lower in Experiment 2, and primacy effects for recall were evident for all conditions. As important, other critical outcomes of Experiment 1 were replicated in Experiment 2. JOLs were influenced by relatedness, even for the between-subjects comparison. Primacy effects were just as large for JOLs as they were for recall, especially for unrelated items that were presented first. oreover, in Figure 2, one should compare JOLs and recall for related items presented first (top two curves, left panel) with JOLs and recall for unrelated items presented first (bottom two curves, right panel). For the first primacy bins (e.g., the bin including Serial Positions 1-3), relatedness influenced recall as expected but had a substantially smaller influence on JOLs. This meaningful interaction illustrates discounting of an intrinsic cue, which is a key effect that would be largely concealed by standard analyses that involve collapsing across serial positions. Experiment 3 In the first two experiments, the intrinsic cue of relatedness in some instances was discounted by JOLs, and the extrinsic cue of serial position was not discounted under some conditions. One reason why the present results may not appear entirely consistent with those from previous research is that the cue of relatedness was manipulated using a blocked-list design. In previous research, the factors under investigation have been presented in a mixed-list design (Connor et al., 1997; Koriat, 1997; Rabinowitz et al., 1982; but see Carroll et al., 1997, who investigated delayed JOLs). In such cases, the effects of intrinsic cues may have overshadowed the effects of extrinsic cues (for detailed discussion of overshadowing, see Price & Yates, 1993). That is, when one uses a mixed-list design, the intrinsic cue of relatedness may be so salient that it consistently has a maximal influence on JOLs, with other cues being less influential when JOLs are made. In terms of the present design, when unrelated items and related items are mixed within the same list, relatedness may overshadow primacy position, resulting in minimal primacy effects on JOLs. To evaluate this possibility, we manipulated relatedness within a mixed-list design. Participants and Design ethod The design of Experiment 3 was a 2 (relatedness) X 2 (first item: begin list with an unrelated item vs. begin list with a related item) mixed-model factorial with item as a within-subject factor and first item as a betweensubjects factor consisting of two groups. Twenty-eight undergraduates from UNCG participated to fulfill a course requirement. Fourteen participants were assigned to each group by order of appearance. aterials and Procedure Lists were constructed for each participant in the following manner. For both groups, related items and unrelated items were presented in alternating serial positions. The list began with an unrelated item for one group and began with a related item for the other group; this factor had no influence on outcomes and hence is not discussed below. The procedure for Experiment 3 was identical to that of Experiment 2 in all other respects. Recall Performance Results Percentage of correct recall performance was computed for each participant separately for related items and for unrelated items. eans across participants' values are reported in Table 1. Recall performance was reliably greater for related items than for unrelated items, t(27) = 13.6.

9 1188 DUNLOSKY AND ATVEY Analyses of serial-position effects were conducted as described above. eans across participants' values are presented in Figure 3. Primacy effects for recall performance were evident in all conditions, which was consistent with outcomes from a 2 (relatedness) X 3 (position) ANOVA. Reliable main effects occurred for relatedness, F(l, 27) = 106.9, = 0.10, and for position, F(2, 54) = 2.6, = 0.04, p =.09. The Relatedness X Position interaction was not reliable, F(2, 54) = 0.01, = Thus, primacy effects were evident and replicate outcomes reported by Brooks (1999). Judgments of Learning eans across participants' values are reported in Table 1. The magnitude of JOLs was reliably greater for related than for unrelated items, t(27) = The serial-position curves for JOLs are presented in Figure 3. A 2 (relatedness) X 3 (position) ANOVA revealed a main effect of relatedness, F(l, 27) = 102.0, = 642.3, and a main effect of position, F(2, 54) = 4.3, = Although the primacy effects appeared larger for unrelated items than for related items (as in Experiments 1 and 2), the Relatedness X Position interaction was not reliable, F(2, 54) = 1.9, = Relative Accuracy of the Judgments of Learning Gammas were computed separately for related items, for unrelated items, and across all items. eans across participants' gam Related items Unrelated items - O Serial-Position Bins Figure 3. For Experiment 3, serial-position curves are presented for recall and for judgments of learning (JOLs) as a function of kind of item (either related pair or unrelated pair), which were presented for study in a mixed-list design. -a mas are reported in Table 3. The trend toward greater accuracy for unrelated items than for related items approached reliability, r(27) = 1.7,/J =.095. Discussion Outcomes from Experiment 3 demonstrate that primacy effects were not overshadowed by the intrinsic cue of relatedness. Also, JOLs did not discount the effects of relatedness in the analyses of JOL magnitude across all items (Table 1). Even so, under some conditions, this intrinsic cue can be discounted by JOLs. For instance, consider the first primacy-position bin (1-3) in Figure 3: Relatedness had a substantially greater influence on recall performance than on JOLs. As in Experiment 2, this key effect is largely concealed when analyses are collapsed across serial position, which highlights the importance of conducting serial-position analyses in evaluating theories of JOLs. General Discussion A major goal of the present research was to empirically evaluate the cue-utilization framework for JOLs (Koriat, 1997). We begin by discussing evidence relevant to this evaluation, focusing on outcomes that suggest the framework will require revision or additional assumptions. We also briefly describe an alternative hypothesis both to illustrate potential limits of the cue-utilization framework and to guide future research. 1 ultiple outcomes of the present research were consistent with the cue-utilization framework. Complementing outcomes from Koriat (1997), we found that item relatedness had a relatively large influence on people's JOLs, both when items were blocked by relatedness (Experiments 1 and 2) and when relatedness was mixed within a list (Experiment 3). oreover, the effect of relatedness was often not discounted by JOLs. The fact that JOLs tracked the effects of relatedness on recall was also manifested in the effects of relatedness on relative accuracy. Namely, relative accuracy was greater when computed across both subsets of items (Table 3) than when computed within a subset of items (e.g., for related items alone). Other outcomes from the present research, however, cannot be readily accounted for by the cue-utilization framework. First, in Experiments 1 and 2, relatedness was manipulated within participants using a blocked-list design, with the order of blocks (consisting of related items vs. unrelated items) counterbalanced across participants. Thus, JOLs for items presented in the first block could not be based on a comparison of the relative recallability of related versus unrelated items across the list. Relatedness still had a relatively substantial influence on JOLs. JOLs also did not greatly discount the effect of relatedness in these between-subjects comparisons, which involved comparing recall and JOLs for related items when they were presented first with the corresponding values for unrelated items when they were presented first. For these comparisons, the effect of relatedness on recall performance 1 Our discussion concerns the intrinsic-extrinsic distinction that was the focus of our research. Thus, our conclusions and alternative hypothesis are not necessarily relevant to the experiential-based influences on JOLs that result from mnemonic cues, which have also been incorporated into the cue-utilization framework (Koriat, 1997).

10 THEORY OF JUDGENTS OF LEARNING 1189 was 30% and 36% for Experiments 1 and 2, respectively, whereas the effect of relatedness on JOLs was 31% and 25% (values were computed from Table 1). These between-subjects effects on JOLs are inconsistent with a core assumption of the framework and pose a challenge for any theory of JOLs. Of course, these outcomes do not indicate that comparisons across items never contribute to the effects of a factor on JOLs but instead demonstrate that comparisons across items are not necessary for observing these effects. Assuming JOLs need not be comparative in nature, it is not obvious how the cue-utilization framework yields the prediction that JOLs will be influenced more by intrinsic than extrinsic cues. In this way, the framework provides more of an empirical generalization (i.e., a taxonomy of effects) and not a theoretical explanation for why various factors differentially influence JOLs. Second, primacy effects occurred for recall performance across most of the present conditions, a finding that replicates research reported by Brooks (1994, 1999). Given that serial position is an extrinsic cue, expectations from the cue-utilization framework were that primacy effects would be minimal on JOLs and that these effects would be discounted. These expectations were not supported. Primacy effects for JOLs were consistently observed across conditions, except for three cases in which floor or ceiling effects constrained interpretation of recall performance or JOLs. oreover, when primacy effects on recall performance were evident, the same magnitude of effects were also apparent for JOLs (but see values for unrelated items in the left panel of Figure 2). Other research from our laboratory demonstrates that people's JOLs can be relatively sensitive to the effects of an extrinsic cue. 2 We investigated how isolating an item within a list would influence JOLs. Participants studied a list of 12 items. Each item was presented individually for 3 s, and immediately after the offset of an item for study, a JOL was made. After all items had been studied and judged, free recall occurred. In the isolated list, 11 items were nonsense syllables, whereas a digit (the isolated item) was in the seventh serial position of the list. In the heterogeneous list, all items were from different categories and the same digit from the isolated list was in the seventh serial position of this list. A critical difference between the digit in the isolated list versus the heterogeneous list was the condition of learning provided by the context of the surrounding items, making isolation an extrinsic cue. As expected, recall was greater for the digit when it appeared in the isolated list (At =71%) than in the heterogeneous list ( = 48%). ost important, people's JOLs did not discount this isolation effect. Namely, JOLs were greater for the digit when it appeared in the isolated list ( = 70%) than in the heterogeneous list ( = 37%). Both the isolation effect for JOLs and the primacy effects described above can be added to a list of other extrinsic cues that influence JOLs in a way that cannot be readily accounted for by the cue-utilization framework (e.g., Begg, Vinski, Frankovich, & Holgate, 1991; Busey, Tunnicliff, Loftus, & Loftus, 2000; Thiede, 1996). By examining primacy effects, we also discovered that even the intrinsic cue of relatedness may be discounted by JOLs. In Experiment 2, consider the comparison of JOLs and recall for related items presented first versus for unrelated items presented first. For the first serial-position bins, recall performance was greater for related items than for unrelated items, yet JOLs substantially discounted this effect. The same pattern was also evident for the first serial-position bins in Experiment 3 in which relatedness was manipulated using a mixed-list design. These effects are inconsistent with the cue-utilization framework. Third, the order in which items were presented influenced JOLs (Table 1). Namely, JOL magnitude was higher when unrelated items were presented first than when related items were presented first, an effect that cannot be readily accounted for by any current theory of JOLs. To help explain these effects, existing theories can be supplemented with heuristics (for heuristic-based hypotheses of JOLs, see, e.g., Connor et al., 1997; azzoni, Cornoldi, Tomat, & Vecchi, 1997). In the present case, the pattern of outcomes may result from anchoring effects in which JOLs made for one class of items are used as an anchor when subsequently making JOLs for another class of items. 3 For items presented first in a list, people may use the middle of the scale for JOLs because "the laboratory setting creates an expectation of an intermediate level of difficulty. The two extremes... are assumed unlikely. Consequently, subjects may anchor on a probability estimate that would reflect intermediate performance" (Keren, 1991, p. 255). Thus, when unrelated items appear first, they will tend to receive relatively intermediate JOLs (e.g., a mean JOL of 40), and the subsequent JOLs for related items will be adjusted upward from this anchor (e.g., an increase of 40 resulting in a mean JOL of 80). When related items appear first, they will also tend to receive relatively intermediate JOLs (e.g., 60), and the subsequent JOLs for unrelated items will be adjusted downward from this anchor (e.g., a decrease of 40, resulting in a JOL of 20). Although this heuristic cannot account for our entire pattern of outcomes, it does provide an explanation of order effects that may be used to supplement more general theories of JOLs. Finally, the effects described above suggest that whether factors are extrinsic cues or intrinsic cues is not fundamental to the differential effects that factors have on JOLs. An alternative to the intrinsic-extrinsic distinction is that a given cue will influence JOLs if individuals believe the cue is diagnostic of recall performance (for knowledge-based hypotheses of JOLs, see Bieman- Copland & Charness, 1994; Dunlosky & Hertzog, 2000; Dunlosky & Nelson, 1994). Consistent with this hypothesis, Koriat (1997) assumed that the effects of intrinsic and extrinsic cues on JOLs are mediated by a person's a priori theories (i.e., knowledge or beliefs) about memory. In contrast to the intrinsic-extrinsic distinction, however, our alternative hypothesis does not indicate that a given cue has special status, and hence the kind of cue per se does not moderate JOL effects. Thus, depending on an individual's knowledge and beliefs about memory, JOLs may be relatively insensitive to some intrinsic cues and relatively sensitive to some extrinsic cues. 2 ore extensive analyses of this research were reported by Dunlosky, Hunt, and Clark (2000, Experiment 1). We report some values here along with inferential analyses because they were not included in the earlier article. 3 Although our data suggest that JOLs are not necessarily comparative in nature, an explanation based on anchoring per se appears to necessitate comparison across items in the list. However, anchoring and adjustment do not require a conscious comparison with an anchor value. That is, basic anchoring effects are evident even when people report being uninfluenced by an anchor and when they are warned about anchoring biases (Wilson, Houston, Etling, & Brekke, 1996).

11 1190 DUNLOSKY AND ATVEY According to this alternative, primacy effects on JOLs are due either to a priori knowledge about the difficulty of learning lists or to developing a belief about task difficulty as learning continues. In either case, an analysis of the task leads to the use of beliefs about the learning conditions when making JOLs (Dunlosky & Hertzog, 2000). Perhaps most important, the between-subjects effect of relatedness has a straightforward (albeit post hoc) explanation. In particular, it seems reasonable that college students have learned that relatedness influences memory, so related items will receive high JOLs regardless of whether unrelated items appear on the list of to-be-remembered items. The intuitive appeal of this explanation is easily illustrated: If your first and last name were a pair on the list of items, you would make a high JOL for this pair even if you could not compare it with any other item on the list. This example also illustrates another potential limitation of the intrinsic-extrinsic distinction. Namely, relatedness is by definition an intrinsic cue; in the example above, however, the relatedness between your first and last name emerges as a joint function of item characteristics and your prior knowledge that is elicited while studying the pair. Even in less extreme cases, semantic relatedness is tied specifically to an individual's prior knowledge. For instance, barre-chord would be highly related for most guitar enthusiasts but would likely be unrelated for those with little or no knowledge of how to play. Our point is that relatedness is not strictly intrinsic to the item pairs. Instead, one's prior knowledge partly defines relatedness, and hence this factor can also be considered an extrinsic cue. The same rationale can readily be derived for other (presumably intrinsic) factors, such as word frequency and imageability. Certainly, Koriat's (1997) intent was to define cues in such a way as to ensure that factors would be reliably classified as either an intrinsic cue or an extrinsic cue. Given that fuzzy boundaries exist between the cues, a factor may not be easily or objectively classified, which would jeopardize empirical evaluations of the cue-utilization framework. Accordingly, future research will benefit from the development of techniques that may be used to classify factors into the intrinsic-extrinsic taxonomy. On a related point, consider EOL judgments and JOLs from Experiment 1. The relationship between the judgments was larger when computed across both related and unrelated items than when computed for only a subset of items (Table 2). This outcome falls directly from the alternative hypothesis described above, assuming that both judgments are influenced by people's beliefs about relatedness. This interpretation has direct implications for classifying a factor as an intrinsic or extrinsic cue because Koriat (1997) offered EOL judgments as a way to classify factors. Although using EOL judgments is objective, if both judgments are based on knowledge about the to-be-classified factor, using EOL judgments to classify a factor as an intrinsic cue may ensure that the factor will also influence JOLs. If so, using EOLs in this way may diminish the likelihood of disconfirming some (but certainly not all) predictions developed from the cue-utilization framework. Unless this problem can be resolved (e.g., by developing other techniques to classify cues), the utility of the intrinsic-extrinsic distinction within the cue-utilization framework will be diminished. In summary, Koriat's (1997) cue-utilization framework was the first general framework of JOLs and motivated our research efforts across three experiments. A central prediction of this framework was consistently confirmed; other outcomes, however, could not be adequately explained. ost notably, item relatedness influenced JOLs even when the analyses were conducted between subjects; primacy effects (an extrinsic cue) on JOLs were apparent both for related items and unrelated items; and JOLs were influenced by the order in which the block of items (related or unrelated items) was presented first for study. We offered both an addition (an anchoring heuristic) and a revision (downplaying the significance of the intrinsic-extrinsic distinction) to this framework so as to guide research directed toward understanding how people judge their memory and learning. References Arbuckle, T. Y., & Cuddy, L. L. (1969). Discrimination of item strength at time of presentation. Journal of Experimental Psychology, 81, Begg, I., Vinski, E., Frankovich, L., & Holgate, B. (1991). Generating makes words memorable but so does effective reading. emory & Cognition, 19, Benjamin, A. S., Bjork, R. A., & Schwartz, B. L. (1998). The mismeasure of memory: When retrieval fluency is misleading as a metacognitive index. Journal of Experimental Psychology: General, 127, Bieman-Copland, S., & Charness, N. (1994). emory knowledge and memory monitoring in adulthood. Psychology and Aging, 9, Brooks, B.. (1994). A comparison of primacy items in free recall. Journal of emory and Language, 9, Brooks, B.. (1999). Primacy and recency in primed free association and associative cued recall. Psychonomic Bulletin and Review, 6, Busey, T. A., Tunnicliff, J., Loftus, G. R., & Loftus, E. F. (2000). Accounts of the confidence-accuracy relation in recognition memory. Psychonomic Bulletin and Review, 7, Carroll,, & Nelson, T. O. (1993). Effect of overlearning on the feeling of knowing is more detectable in within-subject than in between-subject designs. American Journal of Psychology, 106, Carroll,., Nelson, T. O., & Kirwan, A. (1997). Tradeoff of semantic relatedness and degree of overlearning: Differential effects on metamemory and on long-term retention. Ada Psychologica, 95, Connor, L. T., Dunlosky, J., & Hertzog, C. (1997). Age-related differences in absolute but not relative metamemory accuracy. Psychology and Aging, 12, Dunlosky, i., & Hertzog, C. (1998). Aging and deficits in associative memory: What is the role of strategy production? Psychology and Aging, 13, Dunlosky, J., & Hertzog, C. (2000). Updating knowledge about encoding strategies: A componential analysis of learning about strategy effectiveness from task experience. Psychology & Aging, 15, Dunlosky, J., Hunt, R. R., & Clark, E. (2000). Is perceptual salience needed in explanations of the isolation effect? Journal of Experimental Psychology: Learning, emory, and Cognition, 26, Dunlosky, J., Kubat, A., Hertzog, C. (2000). Do age-related differences exist in monitoring learning? Effects of aging on the magnitude and accuracy of quality-of-encoding judgments. anuscript submitted for publication. Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of emory and Language, 33, Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, Hacker, D. J., Dunlosky, J., & Graesser, A. C. (Eds.). (1998). etacognition in educational theory and practice. ahwah, NJ: Erlbaum. Hertzog, C, Kidder, D. P., Powell-oman, A., & Dunlosky, J. (in press). Aging and monitoring associative learning: Is monitoring accuracy spared or impaired? Psychology and Aging.

12 THEORY OF JUDGENTS OF LEARNING 1191 Kelemen, W. L., Frost, P. J., & Weaver, C. A., ffl. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. emory & Cognition, 28, Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Ada Psychologies 77, 2\l-TTi. King, J. F., Zechmeister, E. B., & Shaughnessy, J. J. (1980). Judgments of knowing: The influence of retrieval practice. American Journal of Psychology, 93, Koriat, A. (1997). onitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126, Leonesio, R. J., & Nelson, T. O. (1990). Do different metamemory judgments tap the same underlying aspects of memory? Journal of Experimental Psychology: Learning, emory, and Cognition, 16, Lovelace, E. A. (1984). etamemory: onitoring future recallability during study. Journal of Experimental Psychology: Learning, emory, and Cognition, 10, aki, R. H., & Berry, S. L. (1984). etacomprehension of text material. Journal of Experimental Psychology: Learning, emory, and Cognition, 10, atvey, G., Dunlosky, J., & Guttentag, R. (2001). Fluency of retrieval at study affects judgments of learning (JOLs): An analytic or nonanalytic basis for JOLs? emory & Cognition, 29, azzoni, G., Cornoldi, C, Tomat, L., & Vecchi, T. (1997). Remembering the grocery shopping list: A study on metacognitive biases. Applied Cognitive Psychology, 11, azzoni, G., & Nelson, T. O. (Eds.). (1998). etacognition and cognitive neuropsychology. ahwah, NJ: Erlbaum. etcalfe, J., & Shimamura, A. J. (Eds.). (1994). etacognition: Knowing about knowing. Cambridge, A: IT Press. Nelson, T. O., & Dunlosky, J. (1991). When people's judgments of learning (JOLs) are extremely accurate at predicting subsequent recall. Psychological Science, 2, Nelson, T. O., & Narens, L. (1990). etamemory: A theoretical framework and new findings. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 26, pp ). San Diego, CA: Academic Press. Price, P. C, & Yates, J. F. (1993). Judgmental overshadowing: Further evidence of cue interaction in contingency judgment. emory & Cognition, 21, Rabinowitz, J. C, Ackerman, B. P., Craik, F. I.., & Hinchley, J. L. (1982). Aging and metamemory: The roles of relatedness and imagery. Journal of Gerontology, 37, Schwartz, B. L., Benjamin, A. S., & Bjork, R. A. (1997). The inferential and experiential bases of metamemory. Current Directions in Psychological Science, 6, Son, L. K., & etcalfe, J. (2000). etacognitive and control strategies in study-time allocation. Journal of Experimental Psychology: Learning, emory, and Cognition, 26, Thiede, K. W. (1996). The relative importance of anticipated test format and anticipated test difficulty on performance. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 49A, Thiede, K. W., & Dunlosky, J. (1999). Toward a general model of selfregulated study: An analysis of selection items for study and self-paced study time. Journal of Experimental Psychology: Learning, emory, and Cognition, 25, Wilson, T. D., Houston, C. E., Etling, K.., & Brekke, N. (1996). A new look at anchoring effects: Basic anchoring and its antecedents. Journal of Experimental Psychology: General, 125, Received June 12, 2000 Revision received January 8, 2001 Accepted January 9, 2001