Random error in judgment: The contribution of encoding and retrieval processes. Timothy J. Pleskac, Indiana University

Transcription

1 Random error in judgment 1 Running head: Random error in judgment Random error in judgment: The contribution of encoding and retrieval processes Timothy J. Pleskac, Indiana University Michael R. Dougherty, A. Walkyria Rivadeneira, & Thomas S. Wallsten University of Maryland College Park Address correspondence to: Timothy J. Pleskac, Ph.D. The Department of Psychological and Brain Sciences Indiana University 1101 East Tenth Street Bloomington, IN tim.pleskac@gmail.com Phone: (812) March 12, 2007

2 Random error in judgment 2 Abstract Theories of confidence judgments have embraced the role random error plays in influencing responses. An important next step is to identify the source(s) of these random effects. To do so, we used the stochastic judgment model (SJM) to distinguish the contribution of encoding and retrieval processes. In particular, we investigated whether dividing attention during encoding and/or retrieval changed the consistency of judgments during a statement verification task. Analysis of both choices and confidence ratings revealed: (1) DA at encoding increased random error in the stored information thereby decreasing discrimination ability; and (2) DA at retrieval had no effect on discrimination ability, criterion, or trial-by-trial variability. We discuss the implications of these results in terms of theories of memory and of judgment. (121 words) Keywords: confidence, random error, memory, encoding, retrieval

3 Random error in judgment 3 The judgments people make, whether they are about recent political issues or about last night s baseball game, are often from memory. This linkage creates suppositions that complete models of judgment (as well as memory) require an understanding of how judgment processes interact with memory processes (see Newell, 1973). Yet, memory processes are typically relegated to auxiliary assumptions in theories of judgment (e.g., Erev, Wallsten, & Budescu, 1994; Tversky & Koehler, 1994). Indeed, as our research here shows, memory and judgment researchers alike have much to learn from a tighter integration of these two domains. The purpose of the present research was to examine the implications of divided attention at encoding and retrieval for judgment processes. The implied assumption of our work is that the output of the memory system serves as input to judgment processes. Our conceptual framework, which is detailed in Figure 1, is informed by work using the Stochastic Judgment Model (SJM; Wallsten & González-Vallejo, 1994). The basic idea is that any judgment is dependent on both how information was encoded into memory at the time of study and how information was retrieved from memory at the time of the test. This conceptualization naturally leads to the hypothesis that those variables that affect encoding and retrieval processes, will also affect judgment processes. As the SJM forms the cornerstone of the work presented here, we present an overview of the model next. We then specify how the model can be used for examining the relationship between memory and judgment. The SJM Framework The SJM measures discrimination ability, criterion or response placement, and trial-bytrial variability. To do so, the model is restricted to well-defined knowledge domains where true statements (e.g., Al Pacino celebrates his birthday before Harrison Ford.) and their syntactically

4 Random error in judgment 4 identical complementary statement (e.g., Harrison Ford celebrates his birthday before Al Pacino.) can be randomly sampled. When faced with a particular true statement, s, judges are assumed to use information stored in memory (stage 1 of Figure 1) to form a base confidence in the veracity of the statement, y, with high values of y indicating greater veracity. We assume the same information is used to respond to complementary false statements. 1 Consequently, the distributions over y for true and false statements are set to opposing normal distributions with variances equal to one (see top panel of Figure 2). The distance between the means, d, measures the ability of judges to discriminate between true and false statements. Differences in d could be due to changes in the separation of the distributions, changes in the variance of the unstandardized base-confidence distributions, or both. At test judges probe memory to retrieve their base confidence level that statement s is true (stage 2) forming a covert level of confidence (x). Their confidence is a function of the base veracity level (y) and random error (ε), x = y + ". (1) Formally, ε is normally distributed with a mean of 0 and a variance of s 2 (see middle panel of Figure 2). The random error is due to variables present at the time of the test (such as DA). Judges respond true if and only if x > k. (2) Where k is the location of the criterion (see bottom panel of Figure 2). Substituting Equation 1 into 2 yields the participant s response rule: y + " > k. The criterion, k, represents the judges bias to answer true or false. The fully specified SJM predicts the proportion of responses falling into each of the four categories in Table 1. For more information and details of the model see

5 Random error in judgment 5 Wallsten & González-Vellejo (1994). The power of the SJM model over other signal detection frameworks is that it can offer a measure of trial-by-trial variability or random error in judgment. As we explain next, random error has proven an important variable in theories of probability and confidence judgments and can provide new insights into the characteristics of memory processes. Random error in judgment Erev et al. (1994) showed that random effects can account for the paradoxical coexistence of conservatism and over-confidence in human judgment. An important next step is to identify the source(s) of the random error. In terms of Figure 1 s three-stage framework, most of the work has focused on the stage 3 response process showing inconsistencies in how judges fix their response criteria (Budescu, Erev, & Wallsten, 1997). Random variability can also enter via stage 1 encoding either through direct encoding of error from the stochastic environment (Soll, 1996) or via an imperfect or inconsistent encoding process (Dougherty, 2001). During retrieval, Wallsten, Bender, and Li, (1999) report that the consistency with which judges search memory can also impact error. However, Wallsten et al. did not directly manipulate the retrieval process, leaving unknown the contribution of the stage 2 retrieval process to random error. To this end, one might hypothesize that DA at encoding and at retrieval should degrade the quality of both of these processes. Yet, as demonstrated in the memory literature, DA has asymmetrical effects on encoding and retrieval. DA at encoding and retrieval Investigations with DA are based on the hypothesis that if a primary task requires an attention-demanding process then dividing attention between a secondary task and a primary task should disrupt performance on the primary task (Baddeley, Lewis, Eldridge, & Thomson, 1984). When attention is divided during the encoding process performance is degraded, implying that

6 Random error in judgment 6 encoding demands attentional control (e.g., Craik, Govoni, Naveh-Benjamin, & Anderson, 1996; Naveh-Benjamin, Craik, Guez, & Dori, 1998). In contrast, retrieval processes appears to occur without much attentional control, leading some to characterize it as protected, particularly for recognition (Craik et al., 1996; Naveh-Benjamin et al., 1998). Fernandes and Moscovitch (2000), though, contend that DA s effect at retrieval depends on the nature of the DA and retrieval tasks: DA at retrieval is assumed to affect memory performance if the primary and secondary tasks compete for the same representation system (e.g., verbal representations). Our study examined whether DA might lead to different levels of random error in judgment thereby identifying the contribution of different memory processes to random effects in judgment. To test the effect of DA on judgment, we had participants learn true statements about the order celebrities celebrate their birthdays in a given year. Later they were tested on their memory for the veracity of the learned statements. Judges completed the task with DA while encoding information, retrieving information, or both. We can now state our hypotheses within SJM. For encoding, we expected DA to introduce variability to the base-confidence distributions lowering levels of d as compared to full attention (FA). To further test this hypothesis we collected confidence ratings. Dougherty (2001) showed that confidence ratings can dissociate between changes in the mean distances between unstandardized base confidence distributions and changes in variance. We investigated the effect of DA on retrieval with two different DA tasks (digit monitoring and object monitoring) to test Fernandes and Moscovitch s (2000) prediction that the effect of DA is content specific. We expected object monitoring to compete for the same verbal representation system used to assess the statements and to introduce more trial-by-trial variability into the observed responses than digit monitoring.

7 Random error in judgment 7 Method Participants. Twelve graduate students were recruited from the University of Maryland campus to complete thirteen 45-minute sessions. They were each paid $100 for their participation and were entered into a drawing for an additional $50 with the chance of winning the bonus contingent on their performance. Experimental design. We used a 2 (FA or DA integer-monitoring at encoding) x 3 (DA object-monitoring, DA integer-monitoring, or FA at retrieval) within-subjects statement verification task. Within each condition we collected a sufficient number of observations to estimate the SJMs at the individual level. Statement verification task. The statement verification task was administered with a Java application on a PC. During the task, participants were asked to identify which of two celebrities born in the 20 th century celebrated their birthday earlier in a given year. A list with 3,343 famous people born between 1900 and 2000 and their birthdays was obtained with a $10 donation from the web site: FamousBirthdays.com. Celebrities were randomly paired without replacement, and six sets of 65 pairs and six sets of 100 pairs were then drawn to construct the true statements. Thus, each participant only experienced a pair and a celebrity once during all 12 conditions (6 practice and 6 experimental). During each condition, participants encoded from a computer screen each true statement for 2 seconds. Each statement was shown twice and in the same order, insuring all statements appeared in equally spaced intervals. The order for each repetition was randomly assigned for each participant. Each repetition used a different version of the true statement (e.g., Al Pacino celebrates his birthday before Harrison Ford or Harrison Ford celebrates his birthday after Al Pacino). The order of presentation was randomly chosen.

8 Random error in judgment 8 During the self-paced test stage, participants verified the truth of the statements they had just encoded, and later at an equally spaced interval their syntactic complements. The presentation order of complementary statements was randomly chosen. As in the learning phase, two versions of the true statement and two versions of the false statement were used during the test phase. Each version was randomly assigned and occurred equally often within a condition. When a statement was displayed on the computer screen participants used a mouse with their dominant hand to press a button labeled with the response (True/False) on the screen. Then participants entered their confidence in their choice using a slider at the bottom of the screen. For each statement participants correctly identified, they earned 10 points and lost 10 points if they were wrong. DA tasks. Both DA tasks were administered with a Java application on the same computer. The integer-monitoring task involved listening to a continuous list of integers between 10 and 99 played over headphones at a rate of two seconds per integer. Using their non-dominant hand, participants pressed the spacebar when they heard three even integers in a row. During the DA task and matching Fernandez and Moscovitch s (2000) procedures, for every 50 integers approximately nine sets of three even-integers in a row occurred. Participants earned (lost) five points per correct detection (error). The object-monitoring task matched the integer-monitoring task, except that participants listened to a list of objects and had to indicate when three man-made objects in a row occurred. The list of objects, 45 man-made and 45 natural objects, were randomly selected from Battig and Montague s (1969) category norms. Procedure. The twelve participants were well trained in the experimental conditions. After volunteering, participants were familiarized with the statement verification, DA-integer

9 Random error in judgment 9 monitoring, and DA-object monitoring tasks. Then they completed a baseline condition of each of the DA tasks alone for 5 minutes. Following the initial session, participants then practiced each of the six conditions with 65 celebrity pairings in separate sessions. 2 After practicing all six conditions, participants then completed the six experimental conditions with 100 celebrity pairings in separate sessions. Participants completed a 5-minute filler task between the learning and test phases of each session. The experimental conditions were counterbalanced between participants. Results We tested our hypotheses about the effect of DA on encoding and retrieval at the individual level using nested SJMs. Three subjects were removed from all analyses because accuracy on the statement verification task was either at ceiling or floor for at least one condition, thereby making it impossible to estimate the full SJM model for these subjects. All significant statistical tests are significant at a.05 level unless otherwise stated. Choices during statement verification. Using the four predicted proportions of the response categories shown in Table 1, the likelihood of the data across all six conditions for each individual was estimated (see Wallsten et al., 1999 for the likelihood expressions). The full model has 3 parameters per condition, d e,r, k e,r, and s e,r, where e = DA-Integer or FA at encoding and r = DA-Integer, DA-Object, or FA at retrieval. Thus, with 18 parameters and 18 df the model is saturated. Non-saturated nested models were constructed to test our hypotheses. Figure 3 diagrams a hierarchy of plausible models for this experiment. For example, the discrimination constant model (13 parameters) tests the hypothesis that d varies with DA, d er = d. We tested each restriction using Read and Cressies s (1988) correction on the loglikelihood ratio statistic, which corrects for low nonzero cell counts. The statistic is

10 Random error in judgment 10 asymptotically chi-square distributed with degrees of freedom equal to the difference in the number of parameters for the two models. Table 2 displays for each participant the likelihoods for the models in Figure 3. The p-values compare each nested model to the full model. The model is rejected at the group level (see the bottom row in Table 2) and for five of the nine participants using a p-value of.10. The discrimination varies with encoding model tests the hypothesis that d varies only between encoding manipulations. This model is not rejected for 7 of the 9 participants, but is rejected at the group level. Individual parameter estimates from the full model for participants 8 and 9 reveal that these two participants had one session during which discrimination wavered (d =0.5 or 0.8 as compared to their average d of about 1.1). After removing these 2 participants from the group analyses, the model was not rejected. The error constant model examines the effect of DA on trial-by-trial variability, s 2. The model was not rejected for any of the individual subjects implying DA at retrieval did not influence trial-by-trial variability. 3 Finally, the criterion constant model shows that criterions (k) also remained constant across conditions. Together these results suggest a highly constrained model best describes the data. It allows d to vary between encoding conditions and sets s 2 and k constant across all six conditions (see supported model in Table 2). The parameter estimates under the supported model for each individual are shown in Table 3. 4 For all participants including participants 8 and 9 the estimates support our hypothesis that the ability to discriminate true from false statements is lower when statements are encoded with DA. Retrieval, however, is not completely cost free. Choice latencies revealed that participants took longer to respond with DA at retrieval, F(2,16) = 5.44, MSE = Post-hoc

11 Random error in judgment 11 comparisons showed that conditions with DA-integer (M = 5.4s; F(1,8) = 6.00) and DA-objects (M = 5.4s; F(1,8) = 6.23) at retrieval had longer choice latencies than FA at retrieval (M = 4.6s). DA at encoding did not have a significant effect, nor was the interaction significant. Consistent with the speed/accuracy trade-off (see Luce, 1986), average choice latencies (across conditions) were also negatively correlated with participants trial-by-trial variability (s 2 ) parameter, r =.82. Table 4 lists the correlations between s 2 and the choice latencies for each condition. No other parameter was significantly correlated with latency across or within conditions. Confidence ratings. To assess whether DA at encoding and/or retrieval led to different levels of under/over-confidence, we binned the confidence ratings into 6 probability categories (pc) of 10 (R pc = 50, 60, 100). Then we calculated the confidence scores for each condition, [ ] Conf = 1 # N pc R pc " p( correct pc). N pc Where N is the total number of judgments, N pc is the number of judgments made with category pc, and p(correct pc) is the proportion of correct choices for each category. There were no statistically significant differences in the scores across conditions. In fact, participants were well calibrated in their confidence judgments (M = -0.01; SE = 0.01). Figure 4 plots the average calibration curve across participants and conditions. For each participant and condition we also calculated the calibration measures of slope and scatter (see Yates, 1990). Both measures are calculated after converting judges half-scale responses (confidence in their choice) to full-scale responses (confidence the statement is true). The slope score is the difference between the mean confidence for true items and false items. Dougherty (2001) showed that changes in slope are indicative of changes in the mean difference of the unstandardized base-confidence distributions (see top panel of Figure 2). Scatter is the pooled variance of participants confidence ratings conditioned on whether the items were true or

12 Random error in judgment 12 false. Changes in scatter are associated with changes in the variance of the unstandardized baseconfidence distributions (Dougherty, 2001). The slope score was significantly lower with DA at encoding (M = 55.1) than with FA (M = 63.0), F(1,8) = 5.67, MSE= There was no significant effect of divided attention at retrieval or an interaction. The scatter score reveals that DA at encoding also increased the variability in confidence judgments: Scatter was larger with DA at encoding (M = 187.1) than with FA (M = 148.2), F(1,8) = 6.27, MSE = 3, There was no effect of divided attention at retrieval or an interaction. DA task performance. Figure 5 shows that participants were relatively consistent in their performance on the DA tasks when compared to their baseline conditions. However, a one-way repeated measures ANOVA with 7 levels (one for each DA condition) revealed a significant difference between conditions, F(6, 42) = 3.98, MSE = Post hoc analyses with Tukey s HSD test showed that performance in the DA-object task was significantly less than baseline when participants also had DA at encoding, (q(4,42) = -4.94). 6 Discussion Certainly, to make a judgment from memory people rely on encoding and retrieval processes. The next question then is how do these memorial processes shape judgments and is their impact subject to attentional control? This paper shows that at the individual level the amount of attention judges allocate to encoding information impacts the accuracy and consistency of the encoded information and subsequent judgments. In comparison, the accuracy and consistency of judgments are relatively independent from the amount of attention available during retrieval. These results bear on three interrelated areas: (a) the theoretical understanding of the basic memorial processes of encoding and retrieval, (b) the structure of process models of

13 Random error in judgment 13 confidence judgments, and (c) prescriptive methods for reducing overconfidence. We discuss each next. To be sure, the asymmetrical effect of DA on encoding and retrieval is well documented (e.g., Craik et al., 1996; Naveh-Benjamin et al., 1998). Our results extend these findings beyond traditional measures of accuracy to measures of random error and trial-by-trial variability. The SJM analysis revealed that the protected status of retrieval processes extends beyond the accuracy of respondents to their trial-by-trial variability a result that was replicated nine times at the individual level. DA at encoding, in comparison, decreased judges ability to discriminate true from false statements. Confidence ratings, in turn, gave a more fine grain understanding for this decrement. They attributed the decrease in accuracy to both an increase in the variance of the encoded information and a decrease in the average discriminability of the information. To arrive at this more complete understanding of encoding and retrieval processes we had to use a more complex memory task, one where participants had to recognize both a pair of objects and the relationship between them. This basic task is often used in investigating confidence judgments (e.g., Budescu et al., 1997; Gigerenzer, Hoffrage, & Kleinboelting, 1991), making it an important task to understand the properties of memory processes. Certainly more work is needed in understanding the memorial systems underlying this more complex task. However, given that our results are consistent with results using simpler memory tasks (e.g., Craik et al., 1996), we suspect that it shares many of the same properties as used in simpler memory tasks. Learning the degree to which results from simpler memory tasks generalize to more complex tasks is an important step. In the decision sciences there is a growing interest in developing models of choice and judgment based on principles of cognition rather than axioms

14 Random error in judgment 14 of rationality (e.g., Dougherty, Gettys, & Ogden, 1999; Gigerenzer et al., 1991; Pleskac, in press; Roe, Busemeyer, & Townsend, 2001). Most if not all of these theories assume that information is encoded and retrieved while also assuming that attentional resources play a crucial role in judgments. Our results suggest a need for descriptive process models of judgment to distinguish between the attention demanding encoding process and relatively protected retrieval processes. Furthermore, as the negative correlation between trial-by-trial variability and choice latency indicate, cognitive models of confidence judgments should expand to account for the time course of judgments (e.g., Van Zandt, 2000). Finally, our findings have prescriptive implications for the so-called debiasing of overconfidence. There are several strategies that have been shown to decrease overconfidence, including using an anchoring and adjustment strategy (Arkes, Christensen, Lai, & Blumer, 1987) or a metacognitive strategy to consider reasons why you are wrong (Koriat, Lichtenstein, & Fischhoff, 1980). Our results suggest that the allocation of attention can have a debiasing effect on judgment accuracy, but only during the encoding stage. This conclusion is in stark-contrast to the aforementioned debiasing methods used by Arkes et al. and Koriat et al., whose debiasing attempts have focused on the judgment stage. As work on judgment moves forward, researchers will need to recognize that often biases observed in the judgment literature are the result of predecisional memory processes, and not necessarily error-prone judgment processes.

15 Random error in judgment 15 References Arkes, H. R., Christensen, C., Lai, C., & Blumer, C. (1987). Two Methods of Reducing Overconfidence. Organizational Behavior and Human Decision Processes, 39, Baddeley, A., Lewis, V., Eldridge, M., & Thomson, N. (1984). Attention and Retrieval from Long-Term-Memory. Journal of Experimental Psychology:General, 113, Budescu, D. V., Erev, I., & Wallsten, T. S. (1997). On the importance of random error in the study of probability judgment: Part I. New theoretical developments. Journal of Behavioral Decision Making, 10, Budescu, D. V., Wallsten, T. S., & Au, W. T. (1997). On the importance of random error in the study of probability judgment: Part II. Applying the stochastic judgment model to detect systematic trends. Journal of Behavioral Decision Making, 10, Craik, F. I., Govoni, R., Naveh-Benjamin, M., & Anderson, N. D. (1996). The effects of divided attention on encoding and retrieval processes in human memory Journal of Experimental Psychology: General, 125, Dougherty, M. R. P. (2001). Integration of the ecological and error models of overconfidence using a multiple-trace memory model. Journal of Experimental Psychology: General, 130, Dougherty, M. R. P., Gettys, C. F., & Ogden, E. E. (1999). MINERVA-DM: A memory processes model for judgments of likelihood. Psychological Review, 106, Erev, I., Wallsten, T. S., & Budescu, D. V. (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review, 101, Gigerenzer, G., Hoffrage, U., & Kleinboelting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98,

16 Random error in judgment 16 Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning & Memory, 6, Luce, R. D. (1986). Response Times: Their role in inferring elementary mental organization. New York, NY: Oxford University Press. Naveh-Benjamin, M., Craik, F. I., Guez, J., & Dori, H. (1998). Effects of divided attention on encoding and retrieval processes in human memory: Further support for an asymmetry. Journal of Experimental Psychology: Learning, Memory, & Cognition, 24, Newell, A. (1973). You can t play 20 questions with nature and win: Projective comments on the papers of this symposium. In W. G. Chase (Ed.), Visual information processing (pp ). New York: Academic Press. Pleskac, T. J. (in press). A signal detection analysis of the recognition heuristic. Psychonomic Bulletin & Review. Read, T. R. C., & Cressie, N., A.C. (1988). Goodness-of-Fit Statistics for Discrete Multivariate Data (Springer Series in Statistics): Springer. Roe, R. M., Busemeyer, J. R., & Townsend, J. T. (2001). Multialternative decision field theory: A dynamic, connectionist model of decision making. Psychological Review, 108, Soll, J. B. (1996). Determinants of overconfidence and miscalibration: The roles of random error and ecological structure. Organizational Behavior & Human Decision Processes, 65, Tversky, A., & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101,

17 Random error in judgment 17 Wallsten, T. S., Bender, R. H., & Li, Y. (1999). Dissociating judgment from response processes in statement verification: The effects of experience on each component. Journal of Experimental Psychology: Learning, Memory, & Cognition, 25, Wallsten, T. S., Budescu, D. V., & Zwick, R. (1993). Comparing the calibration and coherence of numerical and verbal probability judgments. Management Science, 39, Wallsten, T. S., & González-Vallejo, C. (1994). Statement verification: A stochastic model of judgment and response. Psychological Review, 101, Van Zandt, T. (2000). ROC curves and confidence judgments in recognition memory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 26, Yates, J. F. (1982). External correspondence: Decompositions of the mean probability score. Organizational Behavior & Human Decision Processes, 30,

18 Random error in judgment 18 Author Note Timothy J. Pleskac, Department of Psychological and Brain Sciences, Indiana University. A. Walkyria Rivadeneira, Department of Psychology, University of Maryland College Park. Michael R. P. Dougherty and Thomas S. Wallsten, Department of Psychology and Program in Neuroscience and Cognitive Science, University of Maryland-College Park. This material is based on work supported by the National Science Foundation under Grant SES awarded to Michael R.P. Dougherty. A National Institute of Mental Health Research Service Award (MH019879) awarded to Indiana University supported the first author while writing this paper. Portions of this paper were presented at the 2004 Annual Meeting of the Society for Mathematical Psychology, Ann Arbor, MI.

19 Random error in judgment 19 Footnotes 1 Empirical evidence supports this assumption of complementarity (Wallsten, Budescu, & Zwick, 1993). 2 During these practice conditions they studied 65 statements with 3 repetitions in each condition. This made the verification task too easy as for all conditions all participants were above 95% accuracy. The increase in studied items and fewer repetitions in the experimental conditions proved difficult enough. 3 In addition to replicating the effect 9 times, the design had adequate power to detect a reasonable effect size in trial-by-trial variability. Simulations using typical parameter estimates found that for each individual our design would have detected a difference between trial-by-trial variability with differences between s 2 of.4 or larger. 4 Criterion estimates are centered so that the halfway point between the two (indifference) is set to 0. 5 Trials on which response times exceeded two standard deviations above the mean were trimmed. This accounted for less than 5% of the trials for the true/false responses. 6 Due to unequal sample sizes within each level and six missing data values from a computer error we used Kirk s (1995) cell means model approach to analyze the data. To increase the normality of the scores, the analysis was done with a log-odds transformation.

20 Random error in judgment 20 Table 1 False Complement True False True Complement True m tt m t,f False m f,t m f,f A table displaying the four possible categories into which a judge s response to a complementary pair of statements can fall. The variable m ij identifies the number of statement pairs that fall into each category. The SJM predicts the proportion of responses falling into each category.

21 Random error in judgment 21 Table 2 Full Discrim. Varies with encoding Discrim. Constant Error Constant Criterion Constant Supported Fully Constrained Parameters Participants L L p L p L p L p L p L p Across Ps Remove 8 &

22 Random error in judgment 22 Above are the log-likelihoods and the p values of the likelihood ratio tests of each constrained model compared to the full model using Read & Cressie s (1988) power divergent statistic. The columns are identified by their respective model and their respective number of parameters in each model at the individual level. The rows labeled with participant numbers test the model at the level of the individual, while the bottom two rows test the model at the level of the group.

23 Random error in judgment 23 Table 3 d DA, d FA, s, k, Mean The SJM maximum likelihood parameters for the supported model.

24 Random error in judgment 24 Table 4. DA-Object Retrieval DA- Retrieval FA Encoding DA -81* -.84* -.81* FA * The correlations between participants trial-by-trial variability parameter (s2) and choice latencies for each condition. Consistent with the speed-accuracy trade-off there is a negative correlation. No other parameters were significantly correlated with choice latencies within or across conditions. Average choice latencies (across conditions) were negatively correlated with participants trial-by-trial variability (s 2 ) parameter, r =.82. * = p<.05

25 Random error in judgment 25 Figure Caption Figure 1. The three hypothesized processes a judge uses to produce a judgment. Figure 2. An illustration of the stochastic judgment model. The top panel shows two overlapping distributions representing the judge s base confidence, Y, in true and false statements. Because the statements are randomly sampled and are complements of each other, the distributions are mirror images of each other with d equaling the difference in their means. Their variances are scaled to 1. The second panel shows the distribution of error present at response. It has a mean of 0, and the variance, s 2, is a parameter representing trial-by-trial variability. The third panel shows the judge s covert confidence, X. It is formed from the judge s base confidence, but is perturbed by error. The judge maps his/her response from his/her covert confidence using the criterion k. Figure 3. A hierarchy of nested SJM models, each testing a particular hypothesis. For example, the models identified as Discrim Constant and Discrim. Varies with Encoding test whether discrimination varies between the experimental conditions. There are many more possible models, but these tested the relevant hypotheses. See Table 2 for likelihood ratio tests of each model at the level of the individual and group. Figure 4. The average calibration curve across participants and across conditions. Figure 5. Participants average response time when responding during the true/false response.

26 Random error in judgment 26 Figure 1 Stage 1: Encode information from the environment into memory Stage 2: Retrieve familiarity information from memory Stage 3: Map a response to covert confidence The three hypothesized processes a judge uses to produce a judgment.

27 Random error in judgment 27 Figure 2 An illustration of the stochastic judgment model. The top panel shows two overlapping distributions representing the judge s base familiarity, Y, of true and false statements. Because the statements are randomly sampled and are complements of each other, the distributions are mirror images of each other with d equaling the difference in their means. Their variance are scaled to 1. The second panel shows the distribution of error present at response. It has a mean of 0, and the variance, s 2, is a parameter representing trial-by-trial variability. The third panel shows the judge s covert familiarity at test, X. It is formed from the judge s base familiarity, but is perturbed by error. The judge maps his/her response onto his/her covert familiarity using the criterion k.

28 Random error in judgment 28 Figure 3 A hierarchy of nested SJM models, each testing a particular hypothesis. For example, the models identified as Discrim Constant and Discrim. Varies with Encoding test whether discrimination varies between the experimental conditions. The number of parameters for each model are in parentheses.

29 Random error in judgment 29 Figure 4 The calibration curve averaged across participants and conditions. Participants were well calibrated and did not show significant differences in over / under confidence across conditions. Though more fine-grain analysis showed systematic differences in their slope and scatter scores.

30 Random error in judgment 30 Figure 5 The average percent correct identification in the DA tasks for each condition. Error bars are standard errors estimated from the MSE of the ANOVA.