Dissociations of Accuracy and Confidence Reveal the Role of Individual Items. and Distinctiveness in Recognition Memory and Metacognition

Transcription

1 Dissociations of Accuracy and Confidence Reveal the Role of Individual Items and Distinctiveness in Recognition Memory and Metacognition Draft for Caren Rotello only Thomas A. Busey Anne Arici Indiana University, Bloomington Submitted Manuscript. Please do not quote. Send Correspondence to: or: Tom Busey Anne Arici Department of Psychology Department of Psychology Indiana University Indiana University Bloomington, IN, Bloomington, IN, Page 1 of 54

2 Abstract In three recognition memory experiments, scale-invariant dissociations between confidence and recognition judgments with novel faces are explored. Composite distractor faces were created by morphing two studied faces. These morph distractors are identified as 'old' more often than studied faces, yet observers give the morph distractors lower confidence ratings when they say 'old'. Similar dissociations are found in forcedchoice experiments, which rule out explanations based on unequal-variance signal detection theory, properties of morphs and differences in typicality. A model based on sampling has properties of a recollective process, and can account for these dissociations. The role of distinctiveness was addressed by varying the typicality of faces, which results in a dissociation in which the highest confidence is associated with the lowest accuracy. Together the results demonstrate the manner in which recognition and confidence judgments depend on relations among faces, and are consistent with a model of memory in which the observer makes contact with individual memory traces. Recognition and familiarity-based processes may both be active, but affect recognition and confidence in different ways. Finally, the metacognitive strategies subjects have adopted with respect to distinctiveness appear to make them overconfidence when viewing distinctive faces, which may arise from the differential contributions of recollection and familiarity to recognition and confidence. Page 2 of 54

3 Over the last three decades, support has grown for a dual process view of memory that includes processes such as recollection and familiarity (reviewed by Yonelinas, 2002). While their existence is not entirely without controversy, efforts have now turned to address the properties of these processes and how variables such as similarity, distinctiveness and typicality might affect how these processes contribute to performance. In this article, we start with a simple question: What can recognition and confidence responses tell us about the role of memorial processes such as recollection and familiarity when participants are asked to recognize novel faces? To begin to answer this question, consider two test items in a recognition memory experiment. The first item, which we term item 1S, has one very strong match to one very similar item in memory. In contrast, a second item, termed item 2W, has two weaker matches to two somewhat similar items in memory. In an old/new recognition paradigm, subjects may label both items 1S and 2W as 'old' items, but do so for different reasons. Item 1S may seem familiar because of its one strong match, while in the case of item 2W the cumulative familiarity from its two weak matches may equal or even exceed the familiarity of item 1S. This could cause subjects to label item 2W as 'old' more often, which is a classic prototype effect (Solso & McCarthy, 1981; Franks & Bransford, 1971; Neumann, 1977). Whether item 1S or item 2W is more likely to be called 'old' depends on the strength of item 2W's weak matches. How these individual items are accessed and how evidence is accumulated across them is a central question in the field of recognition memory, and we will use metacognitive judgments to distinguish between candidate explanations of our results. In the course of this article we will argue that this particular comparison has several benefits, chief among them that it tends to hold recognition performance at similar levels for both items A and B, and thus allows us to demonstrate scale-invariant dissociations between different measures of cognitive performance. These dissociations, we will argue, demonstrate how Page 3 of 54

4 contact between a test item and individual items in memory can differentially affect confidence and accuracy. Models of Memory The class of models known as Global Memory models (e.g. SAM (Raaijmakers & Shiffrin, 1980; 1981, Gillund & Shiffrin, 1984), MINERVA2 (Hintzman, 1986, 1988), CHARM (Eich, 1982; 1985), Todam (Murdock, 1982), REM (Shiffrin & Steyvers, 1997)) combine or sum activations across all items in memory to compute a scalar that represents global familiarity of a test item. Models of categorization and recognition that rely on summed similarity (e.g. the Generalized Context Model, Nosofsky (1986)) share this property of combination over items to produce a single summary score. These models (or at least specific versions of them) suggest that the contents of individual items in memory is not available for comparison, merely the results of a global match that combines information across items in memory. In contrast to summing information across items in memory, several models have implemented a version in which memory is sampled and a single item is recovered for comparison with the test item. This is analogous to a recollective process because in the final step information from one item is singled out for comparison with the test item. Many of the global memory models described above include an optional recall mechanism, in which a sampling and recovery process can be used to recover individual item information from memory. Only a few models have argued for a single sampling process. For example, the SimSample model (Busey & Tunnicliff, 1999) conjoins the sampling mechanism of SAM (Raaijmakers & Shiffrin, 1980; 1981, Gillund & Shiffrin, 1984) with an MDS representation to predict recognition of novel faces. Unlike SAM, SimSample does not repeatedly sample items. The model was able to successfully account for the behavior of very distinctive and very typical faces in a novel recognition paradigm (Busey & Tunnicliff, 1999). The neural net model of McClelland & Chappell Page 4 of 54

5 (1998) shares many features with the REM model, differing only in that a Max rule rather than a sum is used to compute recognition responses. As support for both processes has grown, many articles have addressed the interplay between the two processes within a dual-process framework (Hintzman & Curran, 1994; Jacoby, 1991; Mandler, 1980; Norman & O Reilly, 2003, Reder et al, 2000; Yonelinas, 1994) and more recently quantitative models (Rotello, Macmillan & Reeder, 2004). Support for this approach is also growing from neuropsychological dissociations (Curran, 1999, 2000; Curran & Clearn, 2003; Ranganath, Yonelinas et al, 2003; Yonelinas, Otten, Shaw & Rugg, 2005). Our goal is to address how these two processes may differentially affect confidence and recognition judgments. When and how do recollection and familiarly processes become involved in the recognition process? To return to the example we began with, item 2W may have a higher recognition rate based on the two weak matches to memory, but item 1S may have higher confidence when it is recognized and subjects endorse it as a previously-seen item. Such a result would suggest that confidence and recognition are based in part by different sources of information, and in the process of describing what this information is we may reveal something of the architecture of the memory processes. For instance, the stronger confidence rating may come from a recollective process that is more active for stronger matches. This, plus the increasing role that metacognitive judgments such as confidence, judgments of learning and remember/know ratings have played in memory research have motivated this line of research. Embedding Structure in Face Recognition Tasks We have chosen to address the questions raised above within a face recognition paradigm, for several reasons. First, given their obvious ubiquity and social relevance, faces have (dare we say it) face validity as a stimulus class. Second, novel faces lack the semantic representations possessed by words, which suggests that perceptually-based Page 5 of 54

6 representations such as face spaces (Valentine, 1991a, 1991b; O'Toole, Deffenbacher, Valentin, & Abdi, 1994) are appropriate and allows us to make contact with the literature on face spaces, which includes concepts such as typicality and distinctiveness as discussed below. Third, the perceptual nature of faces allows us to create new faces with known relations to faces on the study list, thus embedding structure in our stimulus set. This is critical to the one strong match/two weaker matches comparison, since we need to carefully balance the strength of the weaker matches to make them approximately equivalent to the single strong match. Finally, the recognition of novel faces have been show to include both familiarity and recollective processes, which provides an opportunity for us to reveal how they make contributions to recognition and confidence for the one strong/two weak comparison. Previous work with faces by Yonelinas (2001) used analyses of receiver operating characteristic (ROC) functions to argue for the contributions of both recollective and familiarity processes in novel face recognition, and the present studies extend this work to address how these processes influence confidence and recognition judgments. The use of images also allows us to vary the distinctiveness of the faces to show how distinctiveness of target and distractor items affects confidence and accuracy in different ways. The use of natural images such as faces allows us to work within a conceptual representation that evolved from categorization research and extended to faces (Nosofsky, 1991, Valentine, 1991a, 1991b; Busey, 1998, 2001; Busey & Tunnicliff, 1998). Faces are viewed as points in a multidimensional space, where distance in this space represents similarity. Faces in the middle of this space are viewed as more similar to other faces and therefore more typical, while faces on the fringe are viewed as more distinctive. The properties of this space, even if they are not fully specified, allow us to work with concepts such as typicality, distinctiveness and similarity. In particular, an average in this space, while not always exactly at the midpoint between to endpoints, has Page 6 of 54

7 the property that it is similar to at least two items in the face space (Busey, 1998) and may be approximately equally similar to these two items. To create faces with two similar neighbors, we turned to morphing procedures, which create a face that is a pixel-based blend of two parent faces that have been warped such that their major features align to a set of average locations. This creates a morphed face that appears similar to both faces, but is still distinguishable from either parent face. It is important to note that this procedure is not without its perils, and we would be the first to admit (see Busey, 1998 for the full arguments) that morphs have drawbacks. For instance, morphs suffer from subtle but noticeable blurring artifacts that make them appear smoother and more symmetric than the two parent faces. Likewise, the geometry of face space dictates that the average between two points in the space will often lie closer to the center of the space than either of the two parents, making the morph appear less distinctive and more typical than the parent faces. Nonetheless, it is possible to design appropriately-controlled experiments that reduce or eliminate these differences. In some cases it may not be desirable to eliminate the differences between morphs and parents as these differences may reveal aspects of the memory systems. We return to these issues in the General Discussion, and we hope to convince the reader that the benefits of our morphs outweigh their deficits, in part because of the carefully controlled experiments they allow, as described next. The morphing procedure creates a morph with known similarity relations to the two parents. We will place the parents in a list of faces to be studied by subjects, and then present the parents and the morph on the test list. A parent face will have one strong match in memory itself. The morph itself was not studied, but its parents were, and so it will have two weak matches in memory. If we choose parent faces that are similar enough to each other, they will generate a morph that will have an approximately equal probability of being labeled as 'old' (i.e. previously seen) as either of the parent faces. Of Page 7 of 54

8 course if we chose two parent faces that were very dissimilar, we may find that the morph is almost never labeled as one previously seen. Choosing two parent faces that are almost identical will produce a morph that is so close to the parents that effectively the morph will have been presented twice on the study list (or at least two faces very similar to the morph will have been presented). This would make the morph seem extremely familiar and cause subjects to label it as 'old' with high probability. These two extremes demonstrate that it is possible for us to vary the similarity of the parents to vary the relation between the parent recognition rate and the morph recognition rate. In fact, if the parents are very similar to each other, subjects may be more likely to say 'old' to the morph (which they haven't seen) than to the parent (which they have). Having said this, our specific goal is to chose the similarity of the parents such that the morphs and parents are labeled 'old' approximately equally by subjects. If we achieve this, we are then in a position to address how metacognitive judgments like confidence are affected by the 1- Strong/2-Weak manipulation while avoiding the somewhat trivial result that higher accuracy items are often associated with higher confidence. At the onset it is reasonable to ask whether balancing the likelihood of saying 'old' to morphs and parents is a good idea. After all, technically the parents are targets and the morphs are distractors, and if subjects are equally likely to say 'old' to both types of stimuli then this implies, under a traditional view of memory, that performance is at chance. There are several responses to this criticism. First, viewing the morphs and parents as traditional distractors and targets is misleading, because we have built in specific relations between the morphs and parents. There are many examples from the memory and categorization literature in which prototype stimuli like the morphs are called 'old' at rates that can exceed the studied items (e.g. Solso & McCarthy, 1981; Franks & Bransford, 1971; Neumann, 1977). This is the classic prototype effect, and different models will make different predictions for the relation between morphs and Page 8 of 54

9 parents. Second, other distractors and targets that are included in the study list can demonstrate that subjects do in fact distinguish between studied and unstudied items, and thus they are not just guessing at all stimuli. Finally, while recognition performance may be equal for morphs and parents, subjects may assign different levels of confidence to the two sets of stimuli, which would imply that they arrive at their decision to say 'old' for different reasons for the two types of stimuli. These reasons illustrate why it would be a mistake to dismiss the data from morphs and parents as uninterpretable based on equivalent 'old' rates. Our particular application of the 1-Strong/2-Weak focuses on the metacognitive judgments that are associated with recognition responses. The traditional approach within signal detection theory is to view confidence as a finer set of criterion placed on some evidence axis that also includes a decision criterion (e.g. Wixted, 1992). Examples of this include models based on signal detection theory and its variants that are applied to receiver operating characteristic (ROC) curves. Yonelinas has argued that highconfidence responses are associated with responding based on a recollective process (Yonelinas, 1997). Thus a focus on the metacognitive responses has proved fruitful in the past when addressing the nature of memorial processes. As we will demonstrate, the 1S/2W paradigm balances recognition rates and thus provides for scale-invariant dissociations of recognition and confidence judgments. Taken together, our results strongly suggest that test items make enough contact with individual traces in memory in order to influence confidence ratings, and illustrate how distinctiveness can affect recognition and confidence judgments in very different ways. Experiment 1 The goal of Experiment 1 is to measure confidence and recognition responses in the 1S/2W paradigm. We generated 8 morphs from 8 pairs of parent faces and embedded these in an old/new recognition paradigm that included 30 faces at study and 60 faces at Page 9 of 54

10 test. All of the stimuli were faces of bald men that have neutral expressions and are cleanshaven. The goal of Experiment 1 is show parent faces at study and then test both the parent faces and the morphs. Parent faces are (obviously) similar to themselves and therefore have at least one strong match to memory when tested. Morphs are less similar to the parents, but there are two parents associated with each morph and thus morphs have two weak matches to memory when tested. Through pilot testing we chose parents that were similar enough that we had a reasonable chance to balance the recognition rates for parents and morphs. We will get recognition and confidence judgments to both stimulus types to address how confidence and recognition are affected by the two stimuli. Experiment 1 consists of two experiments, Experiment 1a and 1b. These two experiments differed only in that participants in Experiment 1a were asked told to respond accurately at their own pace, while participants in Experiment 1b were encouraged to make rapid recognition responses at the cost of a slight increase in error rates. Rapid responding may emphasize a familiarity-based process (Curran, 1999, 2000; Rotello & Heit, 2000; McElree, Dolan, & Jacoby, 1999), and while we see some small differences in the data for the morphs, the two experiments serve primarily as replications, since the results are very similar. Experiment 1a The goal of Experiment 1a was to measure the recognition and confidence for parents, morphs, and filler faces. By examining the relationship between morphs and parent faces, we can see how confidence and recognition vary as a function of the similarity relations between the two types of faces, and use this information to identify possible memory processes at work. When the morphs are distractors, we anticipate that these faces will attract a large number of false alarms, based on their high similarity to the studied parents and many other faces. Morph distractors have no match in memory, and false alarms to these faces could reflect the contribution of a familiarity mechanism. Page 10 of 54

11 Retrospective confidence ratings were taken after each response, which, when conjoined with the recognition rates in a formal model, provide additional information about the nature of the memory processes as described below. Method Participants Participants were 263 undergraduates from Indiana University who participated in the experiment as part of course requirements. Participants were run in small groups with approximately 4-6 subjects participating at a time in 50 total groups. Note that in an attempt to create a robust dataset that could be used to test and reject models, all of our experiments have a large number of participants, in some cases over 500. This is required by our choice of experimental design, which has many different conditions and allows us to compute hit and false alarm rates for individual faces where this information is useful for model testing. The large number of participants should not interpreted as an indication that subjects were not tested properly or that studies were run in a classroom setting. All participants were carefully run in small groups in the laboratory in experiments that lasted between 30 and 60 minutes. Stimuli The stimuli consist of 60 photographs of bald white men (Kayser, 1985). All photographs show faces at the same frontal view and angle, under the same lighting conditions. All the photographs show men who are clean-shaven and have similar expressions. Sixteen parent faces were used to create 8 morphs, and were included in the 60-face total. The photographic images were displayed on a 21-in. Macintosh gray-scale monitor using luminance control and gamma correction from a Video Attenuator and the VideoToolbox software library (Pelli & Zhang, 1991). The background luminance was Page 11 of 54

12 set to 5 cd/m 2. The gray-scale values in the images were scaled to cover the range between 5 cd/m 2 for black to 80 cd/m 2 for white. Data were collected by using one numeric keypad per participant and a single PowerMac computer. Design and Procedure Two study sets of faces were constructed so that each group included 8 parents, 4 morphs and 18 filler faces, for a total of 30 faces. The design is counterbalanced so that one group of participants studies one set of 30 faces (Set A) and the next group studies a similar set of 30 faces (Set B). At test, both sets A and B are presented to measure the hit rates and false alarms to the targets and distractors. Of particular importance, the morphs and associated parents are not all targets at the same time. That is, if the parents are studied, their corresponding morphs are not studied, and instead are included as distractors at test. Conversely, the morphs that are studied have their associated parents included in the test session as distractors. Across subjects, hit rates and false alarm rates are recorded for all faces. At the start of the study session, participants were told to study each face in preparation for a following memory test. The 30 faces were shown one at a time, in randomized order. Each face was presented for 1200 ms, followed by a 2-second delay before presentation of the next face. Immediately after the study phase, participants were tested in an old/new picture recognition paradigm. The 60 faces were presented in random order and participants were asked to identify each as 'old' or 'new', for faces they had studied or had never seen, respectively. Retrospective confidence ratings were taken immediately after each test face. Participants were asked to respond how confident they were in the accuracy of their just previous response and assign a corresponding confidence value from a scale of 1 to 5 (1=0% confident, 2= 25% confident, 3=50% confident, 4= 75% confident, and 5=100% confident). Page 12 of 54

13 P("Old") Filler Faces Experiment 1a- Recognition Morphs and Parents Higher False Alarm Rate to Morphs Than Hit Rate to Parents Mean Confidence When Say Old 100% 75% 50% Filler Faces Experiment 1a- Confidence Morphs and Parents Higher Confidence to Parents as Targets than to Morphs as Distracters 0.00 Targets (Hits) Distracters (False Alarms) Morphs Morphs (Hits) (False Alarms) Parents Parents (Hits) (False Alarms) 25% Targets (Hits) Distracters (False Alarms) Morphs Morphs (Hits) (False Alarms) Parents Parents (Hits) (False Alarms) Figure 1. Experiment 1a data. Left panel: Probability of saying 'old' to different experimental conditions. Right panel: Mean confidence for 'old' responses. Error bars represent ±1 SEM. Results and Discussion Figure 1 shows the results for Experiment 1a. The mean hit rate to the filler faces (any face that is not a parent or a morph) across subjects was 0.659, with a mean confidence rating of 72.25%, which shows a fairly high hit rate and confidence rating for these faces. The false alarm rate to filler faces was low at 0.209, which gives a d' for these filler faces of Thus our subjects could distinguish target faces from distractor faces with fairly high accuracy and appeared to be on task and making an effort to perform well. The critical comparison is between parents and morphs. The parent hit rate was and false-alarm rate was The high parent false alarm rate may be due to the fact that the associated morph was studied as dictated by the design of the experiment. The morph faces provided both a higher hit rate (0.753) and a higher false-alarm rate (0.712) than the parent hit rate. However, despite being more likely to say 'old' to the morph distractor than to the parent targets, participants were more confident when saying Page 13 of 54

14 'old' to the parents than when saying 'old' to the morph distractors. Confidence when participants said 'old' to parent targets is 68.3%, and for morph distractors is 64.1%. This reveals a confidence and recognition inversion between parent targets and morph distractors: In the conditions where participants are more likely to say 'old' (morph distractors), they are least confident when they say 'old'. This effect replicates in Experiment 1b. Inversions of recognition and confidence suggest that several different processes might be at work, influencing recognition and confidence in different ways. One possibility is that recognition relies on the output of a recollective mechanism with the possible contribution from familiarity-based mechanisms (Yonelinas, 1997). Morphs, in particular, may promote responses based on a familiarity mechanism due to their high similarity to parents and other faces. However, confidence may be influenced more by the outputs of a recollective process such that when a recollection process is used, participants may realize this and give higher confidence values as a result (Yonelinas, 2001; Yonelinas, Kroll, Dobbins, & Soltani, 1999). Clearly the morph distractor cannot be recollected because it was not studied, and this might lead to the lower confidence for the morph distractors. Thus, to investigate the possibility of familiarity-based processes at work with the morphs, we designed a condition to selectively enhance a familiarity mechanism. Experiment 1b The goal of Experiment 1b was to replicate the findings in Experiment 1a, while also promoting reliance on a familiarity-based mechanism by decreasing the ability to use recollective processes. To this end, Experiment 1b was identical to Experiment 1a, except for the addition of a speeded response requirement. Past research in the verbal learning domain has suggested that familiarity-based recognition has an earlier onset and shorter time course in comparison to slower recollective processes that involve searches through Page 14 of 54

15 P("Old") Filler Faces Experiment 1b- Recognition Morphs and Parents Higher False Alarm Rate to Morphs Than Hit Rate to Parents Mean Confidence When Say Old 100% 75% 50% Filler Faces Experiment 1b- Confidence Morphs and Parents Higher Confidence to Parents as Targets than to Morphs as Distracters 0.00 Targets (Hits) Distracters (False Alarms) Morphs Morphs (Hits) (False Alarms) Parents Parents (Hits) (False Alarms) 25% Targets (Hits) Distracters (False Alarms) Morphs Morphs (Hits) (False Alarms) Parents Parents (Hits) (False Alarms) Figure 2. Experiment 1b data. Left panel: Probability of saying 'old' to different experimental conditions. Right panel: Mean confidence for 'old' responses. Error bars represent ±1 SEM. memory (Curran, 1999, 2000; Rotello & Heit, 2000). Requiring participants to respond more quickly may force them to rely more on the results of familiarity mechanisms before recollective processes have time to complete. In fact, this manipulation had relatively little effect on the data, producing only slight increases in the morph false alarm rates and confidence when the morph distractor was labeled as 'old'. Method The stimuli, design and procedures were identical to those in Experiment 1a, with the following exceptions: Participants Participants were 209 Indiana University undergraduates who participated in the experiment for course credit. Participants completed the experiment in small groups with approximately 4-6 individuals per group in a total of 50 groups. Design and Procedure Participants were instructed that their reaction times would be recorded for each test response. They were encouraged to respond "old" or "new" as quickly as possible, while Page 15 of 54

16 maintaining a relatively high level of accuracy. There were no questions from participants about the implementation of these instructions. Results and Discussion Figure 2 shows the data from Experiment 1b. Similar to Experiment 1a, this experiment resulted in a high hit rate (0.669) and mean confidence rating (72.25%) for filler faces. The false-alarm rate to filler distractors was low at Parent faces have a hit rate of and false-alarm rate of Of particular interest, this experiment replicated the confidence and recognition inversion for the morphs and parents found previously in Experiment 1a. The morph false-alarm rate of was higher than the parent hit rate (0.682), while mean confidence to morph distractors (62.38%) remained lower than that of the parent targets (70.1%). As with Experiment 1a, participants were least confident with the stimuli (morph distractors) that they were most likely to give an 'Old' rating. The confidence/recognition dissociation seen in Experiments 1a and 1b may arise from several sources, and below we consider several possibilities and the evidence for each. Many of the concerns raised by the special properties of morphs will be addressed in Experiments 2 and 3. First, the 1 Strong/2 Weak comparison that dissociates confidence and recognition suggests that confidence does not combine in the same way that recognition evidence does, could produce 1S<2W for recognition, but 1S>2W for confidence. This suggests the possibility that subjects have access to single traces at a metacognitive level, and are sensitive to the nature of the evidence that is accumulated. As we shall see, the results of Experiment 2 supports this suggestion. Second, several authors have noted that morphs have surface properties that distinguish them from the parent faces (Busey, 1998; Zaki & Nosofsky, 2001). Zaki and Page 16 of 54

17 Nosofksy (2001) used a particularly interesting approach to demonstrate that morphs seem more familiar to observers than other faces. They presented brief flashes of a rectangle in a study list and told participants that they were participating in a subliminal memory experiment. In fact, no faces were shown at study. Participants were then given a standard old/new recognition test where of course all of the stimuli were distractors. Participants gave higher familiarity ratings to morphs than to parent faces. This suggests that the differences previously documented between morphs and parents may contribute to differences in the recognition response, although global typicality remains a possibility as well as discussed below. Busey (1998) demonstrated that in addition to the surface features such as smoothness and symmetry, both of which could lead to a feeling of familiarity, morphs also lie in a denser region of the face space constructed via multidimensional scaling and are considered more typical than parent faces. Either or both of these differences could contribute to the effects reported by Zaki and Nosofsky (2001) if observers are relating the faces to some global face space representing all faces they have learned. Morphs may lie in a denser region and therefore seem more familiar due to greater global summed similarity. Likewise, the smoothness of the morphs' appearance may rob them of the kinds of distinctive features that lead to high confidence responses, which could lead to the confidence/recognition dissociation. Unfortunately the morphing procedure confounds the surface feature artifacts with high typicality and therefore these two effects are difficult to tease apart in the Zaki and Nosofsky design. Regardless, this experiment illustrates that a recognition bias exists for morphs which must be taken into account when considering the relation between recognition and confidence. We address this in Experiment 2. Finally, the high 'old' rates and low confidence may have a statistical explanation when viewed within the context of a decision-based model such as signal detection theory (SDT). The SDT model is not a process model of memory like the global memory Page 17 of 54

18 models, but instead represent a model of the decision and possibly the production of confidence ratings once the output of a memory system has been produced. For example, many global memory models produce a single value as output that represents the strength of the match between a test item and the items in memory. Setting aside the exact mechanism, SDT assumes that some unidimensional value is produced and that this value is compared with a decision criterion and criterion associated with confidence ratings (e.g. Wixted, 1992). Typically as a distribution shifts to the right, more area falls to the right of the decision criterion, giving that condition more 'old' responses. In addition, as the curve shifts to the right it places more of the distribution in the higher confidence regions, thus predicting that observers are more likely to give higher confidence responses when they say 'old' (Wickens, 2002). This provides a tight coupling between confidence and recognition: as a curve shifts to the right, both recognition rates and confidence go up. In its simplest form, this model is incompatible with the kinds of confidence/recognition dissociations seen in Experiments 1a and 1b. However, if one assumes that different conditions have distributions with different variances, this unequal variance version of SDT is quite compatible with the findings of Experiments 1a and 1b. All that is necessary is that the distribution of the morphs be smaller than that of the parents, and that the parent distribution is shifted slightly to the left of the morphs. How these variances are arrived at is not specified; rather these are considered fitted parameters. Thus at this point we cannot rule out the confidence/recognition dissociation as a statistical artifact resulting from distributions with unequal variances. This explanation will be directly addressed in Experiment 2. Experiment 2 Forced-Choice Tests Between Studied and Unstudied Morphs One way to remove the problems associated with the different surface properties and greater typicality associated with morphs is to test morph against morph in a forced- Page 18 of 54

19 choice design. Morph stimuli have certain peculiarities inherent in their design. The benefit of using morphs in these studies is that these photorealistic blends introduce known relations between target and distractor stimuli, which creates a distractor stimulus that might produce a strong response in a familiarity-based process. However, these same benefits could become detrimental if they are not systematically controlled. To control for the morphs characteristics, we modified our experimental design in Experiment 2 to include morphs at both study and test. Figure 3 illustrates our design. Half of the morphs were studied and were therefore targets, while the other half of the morphs had their parents studied and thus became distractors at test. We use a forced-choice paradigm and pair a studied morph against an unstudied morph. This manipulation has two advantages. First, it controls for typicality and global density in memory, since on average the studied and unstudied morphs will have the same overall global density, and thus this account cannot be used as the only explanation of any confidence and recognition differences between studied and unstudied faces. Second, both the studied and unstudied morphs will share the smooth surface features common to morphed images, and so an explanation based on different surface features cannot be used as the only explanation of a confidence/recognition dissociation. The forced-choice design described above is related to the comparison between the morph hit rates and morph false alarm rates from Experiment 1, since when a morph was a distractor its parents were studied, and when a morph was a target it was studied and its parents were not. Recall that the morph hit and false alarm rates were very similar, leading to the possibility that in the current experiment subjects may be equally likely to choose the studied and unstudied morph in the forced-choice design. If this occurs, the critical question is whether confidence will be higher for the studied morph than for the unstudied morph. Such a finding would rule out explanations of the confidence/recognition dissociation based on surface feature, typicality differences, and Page 19 of 54

20 even an explanation based on an unequal-variance signal detection model as describe in the Study Morph Parent 1 Parent 2 discussion of Experiment 2. Method Participants Test vs A total of 502 participants from Indiana University completed this experiment for partial fulfillment of course requirements. Participants were run in small groups of Studied Morph Unstudied Morph Figure 3. Experiment 2 design. Some morphs were studied, while others had just their parents in the study list. At these each critical test trial paired a studied morph with an unstudied morph. approximately 4-6 people, with a total of 90 groups. Procedure The procedure was the same as Experiment 1 with the exception that we compared morphs against each other in a two-alternative forced-choice paradigm. One morph (the target) was shown at study; the other morph (the distractor) was not studied, but its parents were and thus we refer to these two faces as studied and unstudied morphs. Half of the morphs were presented at study, in addition to the parents and filler faces. The other half of the morphs were not studied and served as distractors at test. Neither parent of a target morph was studied. At test, a studied morph was compared against an unstudied morph in a forced-choice test. There were a total of 18 test trials in the forcedchoice recognition test: 4 morph/morph pairs, and 14 target/distractor pairs which were presented in random order. Morphs were paired prior to all data collection and thus the Page 20 of 54

21 two morphs in a pair alternated as target and distractor. The left/right presentation of studied and unstudied morphs was counterbalanced across participants. Results and Discussion For the filler faces (those not studied or unstudied morphs), participants performed at 82.9% accuracy, which corresponds to a d' value of This Mean Conf When Choose Targets (Studied Morphs) Distractors (Unstudied Morphs) P(Choose) Figure 4. Experiment 3 results, with the best fitting linear regression line for studied and unstudied morphs. implies that they were able to distinguish filler target faces from filler distractor faces with reasonably high accuracy. The critical condition in this experiment compares the studied and unstudied morphs. The probability of choosing the studied morph over the unstudied morph is This is clearly not significantly different from.5, and even if it were, it would be in the wrong direction since a number less than 0.5 implies that participants on average would choose the unstudied morph over the studied morph. Thus in head-to-head testing, observers were equally likely to choose the studied or unstudied morphs, which implies that our pilot testing was successful in balancing the similarity of the parent faces such that the unstudied morph seems about as familiar as the studied morph to the subjects. Despite this equal choosing rate, observers were more confident when choosing the studied morph than when choosing an unstudied morph: The average confidence rating when the observers chose a studied morph is 2.11, and when they chose the unstudied morph is This difference is significantly different by a 2-tailed t-test (t(31) = 2.38; p=0.023). Thus we find a further dissociation of confidence and recognition, and in the Page 21 of 54

22 process attribute this dissociation to memorial processes, because we have controlled for visual artifacts associated with the morphing process. This confidence difference also illustrates that subjects were not simply guessing with no contact with the contents of memory. We need to rule out the possibility that this confidence/recognition dissociation is due to just a few faces. Figure 4 shows the scatter plot of the studied and unstudied morphs. Each point represents the choosing rate of a particular morph, graphed against its confidence when it is chosen. There are 16 points of each type because we graph left and right presentations separately. The probability of choosing a particular face is balanced across the graph, but studied morphs are consistently given higher confidence ratings when they are chosen. This occurs even for parents that are chosen only rarely, and thus this effect occurs over the entire range of data, as shown by the linear regression lines fit to the studied and unstudied morph data. The results of Experiment 2 also rule out an unequal variance signal detection model, since the decision axis in a forced-choice task is the familiarity difference between targets and distractors. Even an unequal-variance model will have symmetric, equal variance distributions on this axis, and with equal choosing rates this model has difficulty producing confidence differences. Thus the unequal-variance signal detection model is an unlikely candidate explanation of the confidence/accuracy inversions 1. 1 We conducted a wide range of simulations to verify the inadequacy of the unequal-variance signal detection model as an account of the Experiment 2 data. The challenge is to find a set of parameters that produces equal choosing rates (corresponding to equal areas to the left and right of 0 on the decision axis, yet still have more area in higher confidence regions when the target morph is chosen. We found no solution with gaussian distributions. With extremely asymmetric distributions such as exponential distributions, we found two types of parameters that could produce qualitative patterns analogous to the Page 22 of 54

23 Sampling Model Account of Experiment 2 The confidence/recognition dissociation seen in Experiment 2 is inconsistent with a signal-detection account, including variants in which the distributions of morphs and parents have different variances. This would seemingly contradict any single-dimensional model in which confidence and recognition are both based on the same information. However, there is an alternative model that captures the spirit of a single-dimensional model, yet still might account for the Experiment 2 data. In addition, this model has elements of what might be viewed as a recollective process, and evidence in favor of the sampling model account could be taken as evidence in favor of a recollective process involved in the recognition process of our novel faces. Previously, Busey & Tunnicliff (1999) argued for the existence of a sampling model for face recognition, in which the test face is used to sample faces in memory. Both recognition (the probability of saying 'old') and confidence are related to the similarity between the test and sampled faces. If the two are more similar, the observer is more likely to say 'old' and will be more confident when doing so. The sampling process is related to the similarity between the test face and other faces such that similar faces were Experiment 2 results of equal choosing rates yet higher confidence when studied morphs are chosen. However, neither set of parameters seems reasonable. The first assumes that the distractors have a smaller variance (.8 vs. 1.0), which is consistent with the literature (refs) but suggests that the distractors are more familiar than target items (1.1 vs. 1.0). This latter parameter setting would seem to contradict the Experiment 1 data in which morph false alarm rates were lower than morph hit rates. The second set of parameters that might account for the Experiment 2 data assumes that distractors are less familiar by a factor of.8 but have a much larger variance than target items (1.4 times greater), which again contradicts the ROC findings. These analyses suggest that even the implausible highly asymmetric distributions such as the exponential cannot save the unequal variance signal detection model. Page 23 of 54

24 more likely to be sampled; thus the name SimSample. Although the SimSample model applied the sampling process to faces, many of the model assumptions are Study Morph Parent A Parent B based on the Search of Associative Memory (SAM) model (Gillund & Shiffrin, 1984) and the Generalized Context Model (GCM; Nosofsky, 1986). Both of these models have been extremely successful in accounting for various phenomena in their respective areas, and thus many of the assumptions that underlie SimSample have been validated in previous work. Test More Sim. Studied Morph vs Less Sim. Unstudied Morph Less Sim. Figure 5. Sampling model explanation of Experiment 3 results. The studied morph is more similar to one item in memory (itself) while the unstudied morph is less similar to 2 items in memory (its parents). The SimSample model accounts for a number of effects, most notably the fact that distinctive faces are more likely to be correctly recognized, and distinctive distractors correctly rejected. SimSample is a member of a class of sampling models that can be viewed as process-model instantiations of a recollective process. This results from the fact that the final decision is based on the sampling and recovery of a single item from memory, which is compared against the test item. The sampling model might account for the findings of Experiment 2 according to the following logic, as illustrated in Figure 5. The studied and unstudied morphs were chosen equally often, which implies that in designing the stimuli we were able to adjust the similarity of the parents so that having two nearby parents for the unstudied morph was equivalent to having actually studied the morph in the memory experiment. The sampling Page 24 of 54

25 model accounts for this by having one very similar item to sample for the studied morph, and two somewhat similar faces (the parents) to sample for the unstudied morph. This leads to equal choosing rates in our particular design. However, the sampling model assumes that confidence is based only on the similarity between the sampled face and the test face. Other models such as a summed similarity model would predict that confidence is determined by the summed similarity to all faces in memory, rather than just to the sampled face. This would lead to a close correspondence between confidence and recognition rates, and could not account for the dissociations seen in Experiments 1 and 2. Because the sampling model assumes confidence to be a function of only the similarity between the test and sampled faces, this predicts lower confidence for the unstudied morphs in Experiment 2. This results from the fact that unstudied morphs have a lower similarity to their parents than studied morphs have to themselves. Thus the studied morph will have higher confidence when chosen, which is due to the greater similarity to its own image in memory. The sampling model is not a single-dimensional model in the strict sense, because different sources of information affect recognition and confidence. However, there is only one underlying representation (the face space ). The choosing rates are affected by both the number of nearby faces and their similarity, while confidence is a function only of the similarity between the sampled face and the test face. This leads to a prediction of a confidence/recognition dissociation by the model. The sampling model's account of the Experiment 2 data suggests two conclusions. First, subjects may make contact with individual items in memory, rather than just evaluating a global familiarity value. This supports various recollection-based models in which subjects have access to individual items. Second, that this contact with individual Page 25 of 54

26 items is evidenced by differences in a metacognitive judgment suggests that subjects are aware of the differences enough for it to affect their confidence ratings. The Role of Metacognition in Recognition Memory The factors that affect metacognitive judgments have been well-studied in literature, primarily with verbal materials but also with visual stimuli such as faces. While much of the work on memory models has stressed the nature of the memory representation and the interactions between studied words, test probes and context, the work on metacognition and confidence judgments has stressed the nature of the test item. Metacognition researchers have acknowledged the contributions of the match between the test item and studied items (known as trace access theory, Burke, MacKay, Worthley, & Wade, 1991; Hart, 1967; King, Zechmiester, & Shaughnessy, 1980), but most of the work has addressed those factors at test that might influence confidence judgments beyond trace access. Variables associated with the test probe such as perceptual fluency (Kelley & Lindsay, 1993), ease of encoding (Benjamin, Bjork, & Schwartz, 1998; Begg, Duft, Lalonde, Melnick, & Sanvito, 1989), and test luminance (Busey & Tunnicliff, 1999) have all been shown to selectively influence confidence. These have been formalized by Koriat (1993, 1995, 1997) into what he terms the accessibility hypothesis, which assumes that learners have no knowledge of their own memory beyond what they can retrieve or what they can infer about what they cannot access. Thus, feelings of confidence stem from a combination of the overall accessibility of information, including the completeness, speed, and ease of retrieval. If an item is easily retrieved, the resulting confidence is high. However, a combination of more subtle cues such as partial recall and contextual information can cause a subjective feeling of knowing, even when retrieval is not possible. In summary, models of both memory and metamemory focus on the nature of processes by which information in the test item is used to probe memory. However, Page 26 of 54

27 memory models stress the nature of the representation and the process by which test item information is compared against memory. Several authors have models that have been extended to suggest how confidence might be determined from this memory probe operation (e.g. Clark, 1997; Dobbins, Kroll, & Liu, 1998; Van Zandt, 2000, and Yonelinas, 1994, 2001), but much of the work on face recognition in eyewitness testimony as address the size of the relation and the variables that affect the correlation, rather than taking a formal modeling approach. Sommer, Heinz, Leuthold, & Matt, et a (1995) argued for distinctiveness as a mediator of confidence and ERP waveforms, and distinctive features may be more likely to engage a recollective system. In addition, distinctive features are playing an increasingly important role in models of recognition memory (Nosofsky & Zaki, 2003; Knapp, Nosofsky & Busey, in press), and thus the affect of distinctiveness on confidence needs to be addressed. Distinctiveness may play a role in the Experiment 1 confidence/recognition dissociation, since morphs may elicit high familiarity, yet receive low confidence if confidence depends on processes that examine the number of distinctive features in the face. In Experiment 3 we systematically vary distinctiveness, and find two additional scale-invariant confidence and recognition dissociations. Note that Experiment 3 uses slightly different stimuli and methods than Experiments 1 and 2, but provides converging evidence for the role of distinctiveness in face recognition and metacognition. In addition, Experiment 3 applies a technique called state-trace analysis which the reader may find useful in other contexts, since it provides for scale-invariant dissociations as well. Experiment 3 The role of distinctiveness in recognition and metacognition The confidence/accuracy dissociation found in Experiment 1 has at least three explanations: parents are more distinctive than morphs, they have better-defined texture, and they had one strong match to memory while morphs had two weak matches. Page 27 of 54

28 Experiment 2 controlled for the first two factors while investigating the third by comparing a studied morph against an unstudied morph in a forced-choice test. We found that the one-strong/two-weak manipulation dissociated confidence and accuracy, and thus the different surface properties and typicality of morphs cannot be the whole account of the confidence/recognition dissociation in Experiment 1. The explanation of this dissociation provided by the sampling model still leaves open the possibility that the different properties of the morphs and the distinctiveness of the parents may also contribute to the greater confidence when observers say 'old' to parents in Experiment 1. For example, subjects may have a metacognitive theory of distinctiveness which would lead them to give higher confidence ratings to distinctive faces. If accuracy really is greater for distinctive faces, this metacognitive strategy may be appropriate, but only if subject anticipate how much they should raise confidence. If subjects raised their confidence too much, this could contribute to the high confidence that we saw in Experiment 1 for parents and the low confidence for morphs. Assuming that distinctive faces are better recognized is only a good thing if the observer doesn't overestimate the benefits that distinctiveness provides. In Experiment 3 we explicitly manipulated the distinctiveness of both target and distractor faces in a forced-choice design to address how distinctiveness affects recognition and confidence judgments. Our goal, like that of Experiment 2, is to measure confidence after different conditions have been equated for accuracy. This eliminates the traditional problem that higher confidence items also have the highest accuracy. We constructed sequences of faces that varied in their distinctiveness through a process described in a later section. Figure 6 shows three faces at the four levels of distinctiveness. During the study period, we presented each face at one of the four levels to show at study. This same face (at the same distinctiveness level) was show at test along with a distractor face at one of 4 levels of distinctiveness, including possibly the Page 28 of 54

29 same level as the target face. We adopted the same recognition response and confidence judgment procedure of Experiment 2, which, in conjunction with a fully crossing of target and distractor distinctiveness levels, will demonstrate how target and distractor distinctiveness affects accuracy and confidence. While greater distinctiveness may lead to higher confidence, the essential question of Experiment 3 is whether subjects correctly anticipate how much benefit they will get from distinctiveness. We will rely on an analysis technique known as State Trace Analysis (Bamber, Loftus refs) to demonstrate a dissociation between confidence and recognition that help explain the Experiment 1 results and suggest that subjects vastly overestimate the benefits of distinctiveness in faces. Along the way we will reveal a surprising inversion of confidence and accuracy: Conditions that produce the highest confidence produce the smallest levels of accuracy. To vary distinctiveness, we extended the morphing procedure described previously. Rather than morph two faces, we placed sets of corresponding points on 60 faces and warped them all to a common set of landmarks. We then averaged all of the warped images pixel-by-pixel to produce what we called the 'supermorph'. Similar procedures have been described by Perrett, May, & Yoshikawa (1994) to address the issue of whether average faces are viewed as attractive. The supermorph has many of the properties of a very typical face since it shares attributes with many other faces. It also has some undesirable characteristics such as a very smooth texture, which we address in a later section. This produces a face that not only has very typical feature locations, it also lacks idiosyncratic features such as moles, wrinkles or other marks that would add to the distinctiveness of the face. Thus one way to view the supermorph is that it is an endpoint on a typicality/distinctiveness continuum. To realize this continuum, we created morph sequences between other faces and the supermorph by varying the amount of contribution from the supermorph and the other face. Page 29 of 54

30 This sequence has a variety of properties, including the fact that as the contribution from the supermorph increases, the faces in the sequence will tend to lose distinctive features. We consider this to be a virtue that enables us to address the role of distinctive features in accuracy and confidence judgments. However, to put all faces more or less in the same category, rather than morphing actual faces with the supermorph, we first created a novel set of faces by choosing pairs of the original set of faces to morph together. Thus all of the faces in Experiment 3 reflect some aspects of the morphing procedure. There are two ways to construct the distinctiveness sequence, and each has its benefits and limitations. One involves both feature warping and pixel blending and tends to produce somewhat smooth faces when morphed with the supermorph, as well as very high-contrast caricatures. The other simply warps the features and preserves the same pixel map, which eliminates the blending artifacts that tend to smooth the facial features. We used both sequences and split Experiment 3 into Experiment 3a and 3b. We first describe the methods and results of Experiment 3a and then return to 3b. The major results are consistent with respect to the relation between confidence and recognition. The small differences reveal the role that blending plays on confidence and recognition, and address the contributions of the surface feature differences of parents and morph faces to the confidence/accuracy inversions and metacognitive judgments. Experiment 3a The goal of Experiment 3a is to measure confidence and recognition accuracy at different levels of target and distractor distinctiveness to determine the role distinctiveness plays in confidence and recognition dissociations like those seen in Experiments 1a and 1b. Page 30 of 54

31 BUSEY AND ARICI Figure 6. Sequences of faces that vary in distinctiveness. Each row represents the transformation of one face, with distinctiveness increases for faces on the right. The second face in each series is a morph between two original faces, the face to the left of that is a morph between that face and the supermorph, and the faces to the right are caricatures designed to exaggerate distinctive features in the morphs. Method Stimuli In order to generate faces that vary in distinctiveness, we modified our morphing procedures. We first created 100 morphs by combining pairs of 60 faces from the Heads book. These were all clean-shaven, bald men of different races. These will become the starting point for our distinctiveness series, and we used morphs as starting points to Page 31 of 54