Limits to Iconic Memory 0 Limits to the Use of Iconic Memory Ronald A. Rensink Departments of Psychology and Computer Science University of British Columbia Vancouver, BC V6T 1Z4 Canada Running Head: Limits to Iconic Memory Word count (summary) : 214 Word count (body) : 1788 Author Notes Most of this work was done while the author was with Cambridge Basic Research, Nissan Research & Development, Inc., Cambridge, MA USA. Many thanks to Duncan Bryce, Puishan Lam, Nayantara Santhi, and Monica Strauss for their help in running the experiments. Correspondence should be addressed to R.A. Rensink, Department of Psychology, University of British Columbia, 2136 West Mall, Vancouver BC V6T 1Z4, Canada. (Email: rensink@psych.ubc.ca or rensink@cs.ubc.ca).
Limits to Iconic Memory 1 Limits to the Use of Iconic Memory It has long been known that human vision retains a trace of a stimulus for 300-500 milliseconds after its disappearance. This trace usually referred to as iconic memory has been extensively studied for several decades 1-5. Results to date have led to a belief that it can act as a surrogate, i.e., that as long as it lasts, its contents can be accessed in much the same way as the information in the incoming light. This assumption is directly tested by a full-scan technique based on visual search, which probes iconic memory more directly than the partial report technique traditionally used 1. Results show that iconic memory is indeed a true surrogate, with performance the same as when the stimulus is visible, and with no cost in switching between visible and iconic representations. Results also show that some visual processes are severely limited in their ability to use iconic memory, and can operate only briefly beyond the time the stimulus is visible. These processes appear to involve the individuation of items. It is proposed that this limit is due to the involvement of re-entrant (or feedback) connections from a higher-level area to lower levels of visual processing, and that the full-scan technique can be used to explore the involvement of such connections in various visual processes. The experiments here were all variants of the standard visual search task, where the observer must determine as quickly as possible the presence or absence of a given target among a set of nontarget items (or distractors) in a display. The time needed for this usually increases linearly with the number of items present. Search typically requires 30-60 ms per item; this is thought to correspond to an item-by-item serial scan of focused attention 6. In contrast to the standard design, these experiments examined visual search on displays that were continually interrupted: After a fixed display time (on-time), the display was blanked for a brief interval (ISI or off-time); this was repeated until the
Limits to Iconic Memory 2 observer responded or timed out after 5 seconds (Fig. 1). The blank fields and display backgrounds were the same medium gray, resulting in a continual flickering of the items. Since the stimuli were not always visible, observers needed to use both visual input and iconic memory to carry out the task efficiently. The question examined here is whether the slope (time per item) found using this full scan technique is affected when observers use iconic memory for different fractions of the display cycle. Each condition examined three timing patterns, or cadences: a base cadence of 80/120 (80 ms on; 120 ms off), plus cadences of 80/240 and 200/120, created by increasing the off-time and on-time by 120 ms, respectively. Twelve observers participated in each condition, with the order of cadences counterbalanced. Lighting level was low, but sufficient to allow colour to be easily seen. Each display contained 2, 6, or items, with the target present on a randomly-selected half of trials. Observers were instructed to maintain fixation between trials and were encouraged to keep error rates low; responses were given by pressing one of two response keys. All observers completed 4 sets of 60 trials in each condition. 7 Experiment 1 examined search based on orientation. Condition 1A used a fixed target: a black vertical line of length 0.8, with the distractors (nontargets) similar lines oriented ±30 to the vertical (Fig. 1a). In completely static displays, search for such stimuli generally requires 30-60 ms/item 6. As can be seen from Fig. 1a, similar slopes were found here. No effect of cadence was found (F(2,) = 0.006; p >.9). Thus, search is not greatly affected by the flickering itself, and the information in iconic memory apparently survives without serious degradation for at least 240 ms. The results also indicate that the switch between visible and iconic inputs takes little or no time, and that attentional scanning has the same speed in both. This supports the idea that iconic memory is a surrogate of the visible input, with its contents accessed in much the same way. --------- Place Figure 1 about here ---------
Limits to Iconic Memory 3 Condition 1B used the same type of items, with approximately equal numbers of verticals and lines tilted counterclockwise by 30. The target now was the item that changed orientation between displays (Fig. 1b). In contrast with Condition 1A, a strong effect of cadence was found (F(2,11) = 33.1; p <.0001). In particular, search slowed with increased off-time (p <.001), but not increased on-time (p >.3). Recasting slopes in terms of the number of items held across the temporal gap 8, the reverse pattern was found: no difference in hold with greater off-time (p >.05), but a significant increase with greater ontime (p <.003). These results indicate that performance is a function of on-item alone. More precisely, comparison of slopes shows that search takes place during on-time plus an additional component of duration d=112 ms, with standard error ±19 ms 9. Beyond this limit, iconic memory apparently cannot be used, even though Condition 1A indicates that sufficient information is available with these kinds of stimuli and with these cadences. This suggests the existence of processes that are photo-driven, i.e., operate only during the time that the visible stimuli are present, along with a component that lingers briefly after they vanish. To determine the generality of this effect, Experiment 2 examined flicker search for other changes (Fig. 2). Condition 2A examined whether iconic memory might be used for a longer duration if the changes were sufficiently large. Items here were rectangular outlines 0.4 x 1.2, and targets changed orientation 90 between vertical and horizontal. Search speeds again depended on cadence (F(2,11) = 16.7; p <.0001), with search slowing for increased off-time (p <.0005) but no significant effect for increased on-time (p >.05). Usable duration d was 112 ms ±26 ms, much the same as before. --------- Place Figure 2 about here --------- Condition 2B examined the case of location change. Here, the target moved back and forth by 1.2 each alternation, with distractors remaining stationary. Speed once again
Limits to Iconic Memory 4 depended on cadence (F(2,) = 12.2; p <.0005), with search slowing for increased off-time (p <.001) but no effect found for increased on-time (p >.2). Usable duration d was 118 ms ± 34 ms. Condition 2C tested change in shape, with the target alternating between a circle and a square. Although much more difficult than the other conditions, similar results were found. Slope again depended on cadence (F(2,11) = 11.2; p <.0005), with search slowing for increased off-time (p <.006) but no effect for increased on-time (p >.7). Usable duration d was 136 ms ±36 ms. Condition 2D examined changes in contrast polarity (black vs. white). Speed again depended on cadence (F(2,11) = 7.9; p <.003). Speed slowed marginally with increased off-time (p =.05), and sped up with increased on-time (p <.05). The latter effect has been found previously, where it was taken to indicate a grouping process for contrast polarity that occurs over the course of several hundred milliseconds 8. Comparing the 80/240 and 200/120 conditions (which equates time per alternation) shows that the condition with greater on-time was faster (p <.005). The relative speeds at these two cadences indicate a usable duration d of 140 ms ± 45 ms, a value comparable with the estimates found for the other kinds of change. Experiment 3 investigated the relevant factor for this limit on the use of iconic memory. One possibility is that this exists whenever the task is difficult. To examine this, Condition 3A asked observers to detect a fixed target with a horizontal bar located only slightly higher than those of the distractors. Search speeds were comparable to those encountered in Conditions 1B, 2A, 2B, and 2D (Fig 3a). However, no dependence on cadence was found (F(2,11) =0.79; p >.4). --------- Place Figure 3 about here --------- Condition 3B investigated an identification task, where observers were asked to report the orientation of a T-shaped target (left or right) among a set of L-shaped distractors (Fig 3B). A dependence on cadence now reappeared (F(2,11) =11.9; p <.0003),
Limits to Iconic Memory 5 with search slowing for increased off-time (p <.001) but no effect found for increased ontime (p >.6). Usable duration d was 175 ms ± 26 ms. To check whether the relevant factor in Condition 3B might have been the existence of multiple kinds of possible target, Condition 3C asked observers to detect a T-shaped target among L-shaped distractors; all items could be in any of 4 orientations (Fig 3C). No significant dependence on cadence was found (F(2,11) =2.22; p >.3). Meanwhile, to determine if the left-right distinction might have been relevant, Condition 3D asked observers to detect an L-shaped figure with the stem facing left among similar figures with stems facing right (Fig. 3D). Again, no dependence on cadence was found (F(2,11) =0.74; p >.4). This supports the view that the key factor in the memory limit in Condition 3B was the type of the task, viz., ascertaining some property (direction) about a detected target. Taken together, these results confirm that iconic memory can be used by all visual processes for about 120 ms past stimulus offset. And they confirm that information lasts for at least another 120 ms, and so should be available for any task. However, this cannot always be done. The key factor appears not to be the difficulty of the task or the information required. Instead, it appears to be the nature of the process itself: detecting a fixed item can use iconic information over a duration of at least 240 ms, whereas detecting a change or identifying a fixed item cannot. Iconic memory is believed to have two components: a brief, high-density visible persistence that supports visual integration at relatively peripheral levels, and a longerlasting informational persistence that is mediated more centrally 3,. Visible persistence lasts for about 0 ms from stimulus onset 11, far less than the 200-320 ms differences in onsets used here; in accord with this, items here were always seen as flickering and not as fused. Thus, the relevant memory component is not visible persistence, but informational persistence. But why should there be a limit to its use? And why only for some processes? An interesting possibility involves the establishment of re-entrant (or feedback) connections
Limits to Iconic Memory 6 from higher-level visual areas to lower-level ones. Fixed patterns can be detected by neurons in high-level areas such as IT; these have inputs from much of the visual field, leading to a considerable degree of spatial invariance 12,13. However, to see a particular item change requires that it be individuated, i.e., treated as a particular individual with a particular location. This must be done by linking the spatially-invariant representation with the part of the lower-level spatiotopic map corresponding to the item s location. It has been suggested that such a link could be established by correlating the downward, spatially-diffuse signals from higher levels with the upward, spatially-precise signals from striate cortex 14. But this can only be done reliably as long as reasonably precise information is available from the incoming image; once this is gone, the process must halt. Activity in the striate neurons of macaques persists for somewhat less than 0 ms past stimulus offset 15 ; assuming a similar phenomenon in humans, this would give rise to a brief trace persistence beyond stimulus offset that could account for at least part of the usable durations encountered here. In this view, photo-drive would be a distinctive behavioral marker of a re-entrant visual process, with the limit on the use of iconic memory reflecting the extent to which a process is dominated by the establishment of re-entrant links. In particular, the relatively longer usable duration found for the identification task indicates that the establishment of re-entrant links does not play as dominant a role as it does in the forms of change detection examined here. More generally, by measuring the limits for various processes, some insight may be gained as to the types of components that exist, and the extent to which these involve re-entrant connections.
Limits to Iconic Memory 7 Notes and References 1. Sperling G. (1960). The information available in brief visual presentations. Psychological Monographs, 74: 1-29. 2. Averbach E & Coriell AS. (1961). Short-term memory in vision. Bell Systems Technical Journal, 40: 309-328. 3. Coltheart M. (1980). The persistences of vision. Philosophical Transactions of the Royal Society B: Biological Sciences, 290: 57-69. 4. Long GM. (1980). Iconic memory: A review and critique of the study of short-term visual storage. Psychological Bulletin, 88: 785-820. 5. Haber RN. (1983). The impending demise of the icon: A critique the concept of iconic storage in visual information processing. Behavioral and Brain Sciences, 6: 1-54. 6. Treisman A., & Gormican S. (1988). Psychological Review. 95: 15-48. 7. Search slopes for each observer were calculated by determining the mean response time for each set size, and calculating the least-squares fit through these points. Analysis was based on the average for those observers with a slope greater than ms / item in the base condition. (If search were too fast, it would not use much of iconic memory.) Analysis used either repeated-measures ANOVAs, or paired, twotailed t-tests. Target-present slopes were the basis of analysis; target-absent slopes either indicated the same pattern, or showed no strong effects. Error rates for target-present conditions either followed the pattern of the slopes or showed no strong effects, indicating that speed-accuracy trade-offs were not a factor. Errors in the target-absent condition were generally low (below 2%), and did not vary much over the various conditions tested. No significant differences in baselines were found, except for the base cadence condition of Experiment 2C, which was significantly higher in the base condition (by 0 ms) than 200/120 condition. 8. Rensink RA (2000). Visual search for change: A probe into the nature of attentional processing. Visual Cognition, 7: 345-376. 9. Calculation of usable memory duration d. Assuming that speed is constant in both visible and iconic components (which seems to be true of the conditions studied), the time spent on each alteration is proportional to the fraction of time in the visible component, plus the usable part of the iconic component. Assuming that the time usable in all conditions is at least approximately 120 ms (supported by the results of the analysis), the average of these two speeds provides an estimate of the speed for 0% coverage. More precisely, the usable fraction f in the 80/240 cycle is (80+d) / 320. Assuming the same speed for visible and iconic inputs (supported by the results of Experiment 1A), the fraction f can be measured by the ratio f = s L /s H, where s L is the slope for 0% use (80/120 and 200/120 conditions) and s H the slope for the 80/240 condition. Rewriting, d = 320f 80. The standard error of the mean of d can also be determined from this formula, via the standard errors of the slopes.
Limits to Iconic Memory 8. Loftus GR & Irwin DE. (1998). On the relations among different measures of visible and informational persistence. Cognitive Psychology, 35: 135-199. 11. Di Lollo V & Bischof WF. (1995). Inverse-intensity effect in duration of visible persistence. Psychological Bulletin, 118: 223 237. 12. Felleman DJ & Van Essen DC. (1991). Distributed hierarchical processing in primate visual cortex. Cerebral Cortex, 1: 1-47. 13. Bullier J. (2004). Communications between cortical areas of the visual system. In LM Chalupa and JS Werner, eds, The Visual Neurosciences, Vol 1. p. 522. Cambridge MA: MIT Press. 14. Di Lollo V, Enns JT, & Rensink RA (2000). Competition for consciousness among visual events: The psychophysics of reentrant visual processes. Journal of Experimental Psychology: General, 129: 481-507. 15. Maunsell JHR, & Gibson JR (1992). Visual response latencies in striate cortex of macaque monkey. J. Neurophysiology, 68: 1332-1344.
Limits to Iconic Memory 9 Acknowledgements: This research was supported by Nissan Motor Co., Ltd., and the Natural Sciences and Engineering Research Council of Canada (NSERC). No competing financial interests.
Limits to Iconic Memory Figure Captions FIG. 1 Setup and slopes for Experiment 1. (a). Detection of fixed orientation. Target is a vertical line; distractors were lines tilted ±30. Slope for base cadence (24.3 ms/item) is affected neither by an increase in off-time (24.4 ms/item) nor by an increase in on-time (24.8 ms/item). Since these are target-present slopes from a self-terminating search, search speeds can be obtained by multiplying by 2. The resultant speeds are about 50 ms/item, similar to estimates found elsewhere 6. (b). Detection of changing orientation. Approximately 50% of lines in each display are vertical, and 50% are tilted by 30 counterclockwise. Target is the item that changes between vertical and tilted; distractors are those items that maintain a constant orientation. Slope for base cadence (53.0 ms/item) is strongly affected by an increase in off-time (93.5 ms/item) but not by an increase in on-time (59.3 ms/item). Error bars indicate standard error of the mean. FIG. 2 Slopes for Experiment 2. (a). Detection of changing orientation. Slope for base cadence (42.7 ms/item) is strongly affected by an increase in off-time (78.1 ms/item) but not by an increase in on-time (51.1 ms/item). (b). Detection of changing location. Slope for base cadence (36.0 ms/item) is strongly affected by an increase in off-time (62.9 ms/item) but not by an increase in on-time (41.7 ms/item). (c). Detection of changing shape. Slope for base cadence (75.2 ms/item) is strongly affected by an increase in off-time (113.1 ms/item) but not by an increase in on-time (77.3 ms/item). (d). Detection of changing polarity. Search in base cadence (45.1 ms/item) is not significantly affected by an increase in off-time (57.0 ms/item) but is sped up by an increase in on-time (39.2 ms/item). Error bars indicate standard error of the mean.
Limits to Iconic Memory 11 FIG. 3 Slopes for Experiment 3. (a). Detection of offset horizontal line in target. Slope for base cadence (52.1 ms/item) is affected neither by an increase in off-time (57.7 ms/item) nor by an increase in on-time (52.3 ms/item). (b). Identification of T-shaped item. Slope for base cadence (47.4 ms/item) is affected by an increase in off-time (59.8 ms/item) but not by an increase in on-time (48.1 ms/item). (c). Detection of T-shaped item. Slope for base cadence (31.3 ms/item) is affected neither by an increase in off-time (34.3 ms/item) nor by an increase in on-time (30.7 ms/item). (d). Detection of left vs right. Slope for base cadence (38.8 ms/item) is not significantly affected by an increase in off-time (46.3 ms/item) or by an increase in on-time (41.6 ms/item). Error bars indicate standard error of the mean.
T: D: T: D: time time 40 80 / 120 90 80 / 120 slope (ms/item) 30 80 / 240 200 / 120 70 80 / 240 200 / 120 20 50 error (%) SOA (ms) SOA (ms) (a) Unique Orientation (b) Orientation Change Figure 1
T: D: T: D: T: D: T: D: 80 70 120 70 slope (ms/item) 60 50 80 50 40 30 40 30 error (%) (a) Changing Orientation (90 ) (b) Changing Location (1.2 ) (c) Changing Shape (d) Changing Polarity Figure 2
T: D: T: D: T: D: T: D: 70 70 60 60 slope (ms/item) 50 50 40 40 30 30 20 20 error (%) (a) Detection: Horizontal Offset (b) Identification: T vs L (c) Detection: T vs L (d) Detection: Left-Right Figure 3