Why Pilots Miss the Green Box: How Display Context Undermines Attention Capture

Transcription

1 THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 14(1), Copyright 2004, Lawrence Erlbaum Associates, Inc. Why Pilots Miss the Green Box: How Display Context Undermines Attention Capture Mark I. Nikolic, James M. Orr, and Nadine B. Sarter Institute for Ergonomics The Ohio State University Visual displays often employ the onset or flashing of an element to notify users of important events. Recent research findings and operational experiences in data-rich, event-driven domains, such as aviation, suggest that this design approach, which was supported by findings from early basic research on attention capture, is not always successful. The goal of this study was to examine how display context affects the effectiveness of abrupt onset signals. Participants in this study performed an externally paced visual task while trying to detect abrupt-onset stimuli, which were presented against 5 different display backgrounds and at 2 different eccentricities. The display background varied in terms of its dynamics and its color similarity to the target. Color similarity, the movement of background elements, and increasing target eccentricity resulted in reduced detection performance. The findings from this study help explain why pilots on modern flight decks sometimes miss changes in the status and behavior of their automated systems. More generally, they illustrate the importance of considering display context and the need to adapt findings from laboratory research when designing interfaces for complex environments. Operators in highly automated, data-rich, event-driven domains, such as aviation, are required to track many concurrent, and sometimes unexpected, changes and events. Because of limited attentional resources and an often large amount of available data, these operators could benefit considerably from more effective external attention guidance. Display designers routinely use the onset or flashing of a dis- Requests for reprints should be sent to Nadine B. Sarter, Institute for Ergonomics, 210 Baker Systems Building, 1971 Neil Avenue, Columbus, OH sarter.1@osu.edu

2 40 NIKOLIC, ORR, SARTER play element to capture pilots attention. However, recent findings in the attention literature and a considerable body of operational evidence show that this approach is not always successful. For example, pilots on modern flight decks sometimes miss unexpected changes in the status of their automated flight deck systems (e.g., Abbott et al., 1996; Sarter & Woods, 1997, 2001). These transitions are typically indicated by the onset of a green outline box around a corresponding alphanumeric mode indication on the primary flight display (PFD). Although the use of such abrupt-onset signals for capturing attention in a reflexive bottom-up fashion is widespread and supported by early findings in the visual attention literature (e.g., Jonides, 1981; Jonides & Yantis, 1988; Yantis, 1996; Yantis & Hillstrom, 1994), more recent research has shown that top-down processes play a considerable role in modulating attention capture and guidance (for an overview see Pashler, Johnston, & Ruthruff, 2001). For example, if a person s attention is locked onto a particular location or object, then the onset of signals in different locations is less likely to attract attention a phenomenon called inattentional blindness (Rensink, O Regan, & Clark, 1997; Theeuwes, 1991). Also, expectations of a particular type of signal, such as onsets, offsets, or color changes, will increase the likelihood of that particular cue to capture attention. In other words, visual onsets per se do not necessarily capture attention. Instead, the likelihood of detection depends on the match between a person s active attention control settings and properties of the appearing signal a phenomenon called contingent orienting (Folk, Remington, & Johnston, 1992; Pashler et al., 2001). Top-down cognitive control is one important but not the only contributor to observed problems with using abrupt onsets for capturing attention. Bottom-up influences, such as display context, can play a role as well, but have received little attention in most earlier research, which has been conducted in relatively spartan laboratory environments, involving fairly simple displays and single-target detection tasks (e.g., Yantis, 1996; Yantis & Egeth, 1999; Yantis & Hillstrom, 1994). It is not clear that findings from this research scale up to real-world data-rich environments such as the flight deck, where operators are faced with complex and highly dynamic displays. Typically, these integrated displays contain many elements of varying hues and behaviors. In fact, there is limited evidence from a small number of studies showing detection performance costs in cases where irrelevant distractors were included in a target detection task display (e.g., Folk, Remington, & Wright, 1998). In particular, same-color distractors had a negative effect on target detection, whereas unique but task-irrelevant distractors of a different color did not capture attention. Also, research on change blindness has demonstrated that people are surprisingly poor at noticing the appearance of, or even large changes to, objects when these occur simultaneously with global or local disruptions of a scene (for an overview of findings in this area, see Simons, 2000). Therefore, at any given moment, a potentially relevant cue can be embedded within other ele-

3 ATTENTION CAPTURE AND DISPLAY CONTEXT 41 ments that can mask the onset of that cue due to color similarity or surrounding motion. In addition to display context, cue location must be considered when designing for data-rich domains that are characterized by multiple and distributed information sources. Although cues can be detected at peripheral eccentricities of up to 50 of visual angle (Rinalducci & Rose, 1986), a number of studies have shown that the presence of a demanding central task effectively narrows or tunnels the functional field of view, making it more difficult to extract information from the periphery (Chan & Courtney, 1993; Rinalducci, Lassiter, MacArthur, Piersal, & Mitchell, 1989; Williams, 1985). Therefore, detection rates for signals located further from the visual focusofataskwouldbeexpectedtodecreaseasthevisualangleincreases.asanoperator s visual focus changes, as it often does in data-rich, multidisplay work contexts such as aviation, the visual field is correspondingly redefined, and the changes in visual angles between current focus and a relevant display cue can be large. The goal of this study was to examine the effectiveness of visual onsets for capturing attention when those onsets are immersed in dynamic, multicolor, and data-rich displays, similar to those encountered in real-world domains. In particular, the effects of color similarity, movements of surrounding display elements, central task difficulty, and target cue eccentricity were examined. The findings from this study help explain and address observed breakdowns in mode awareness on modern flight decks and in other work environments and, more generally, they contribute to a better understanding of mechanisms involved in data-driven attention capture. METHODS Participants Sixteen students (9 men and 7 women) from the Ohio State University participated in the study. Participants ranged in age from 19 to 31 years (M = 21.56, SD = 3.03). All participants had prior experience with the puzzle-completion task used in this study. Participation in the experiment was voluntary, and individuals received a compensation of $10 per hour. Displays and Tasks Participants were seated in front of two 21-in. monitors that were placed side by side. A puzzle-completion task (Version of X-Windows Generic Tetris) was presented on the left monitor and served as the primary task (see Figure 1). The object of the game is to complete as many rows as possible with blocks of varying shape that

4 42 NIKOLIC, ORR, SARTER FIGURE 1 Puzzle-completion task (left) and peripheral target detection display (right). The targets (filled boxes) appeared at two locations on the right screen. Only one target was presented at any given time. fall sequentially from the top of the game window. Players are required to rotate and position the falling pieces to create solid rows of blocks at the bottom of the game window. Once a solid row is created, the row disappears and the remainder of the rows shift downward. When rows are left incomplete, the blocks continue to stack vertically until there is no longer any space for new blocks to fall and the game ends. This task was chosen because it is externally paced and visually demanding, thus resembling the tasks of operators in domains such as aviation or process control. Participants secondary task was the detection of targets that appeared on the right monitor at either 35 or 45 of visual angle (see Figure 1). These target locations are greater than those normally found in abrupt onset research, yet they represent operationally relevant cue locations found in applied contexts such as the glass cockpit. Targets were solid green boxes that were presented for 10 sec against one of five different backgrounds: control, monochrome-static, monochrome-dynamic, color-static, and color-dynamic. The baseline control condition consisted of targets appearing against a solid black background. In all other conditions, the targets were embedded in several round-dial gauges and number displays (see Figure 1). In the two dynamic conditions, the needles in the dials were moving continuously and the numbers randomly changed without interruption. Both the frequency of continuous oscillation of the needles and the frequency of random number changes were approximately 1 Hz. In the two color conditions, all display objects, except the round dial gauges (due to limitations of the simulation), were the same color as the target (green), whereas in the monochrome conditions all background elements were shown in white. Note that the targets in this study (solid boxes) were more luminant and thus potentially more salient than the green outline box around the flight mode annuncia-

5 ATTENTION CAPTURE AND DISPLAY CONTEXT 43 tions on the PFD of a regular automated flight deck. They were unexpected in the sense that participants could not predict when, or at which of the two positions, the target would appear. A Silicon Graphics Octane workstation was used to run both tasks. A keyboard was used for performing the primary task with the left hand, and participants pressed the left or right mouse button with their right hand to indicate detection of a near (left button) or far (right button) target. An experimenter was present to observe participants and ensure that their visual focus remained on the Tetris task on the left monitor. Experimental Design The study employed a factorial within-subjects design. The four factors were: (a) color of the distractor elements in the target detection display (monochrome white or color), (b) presence of moving elements in the display (static or dynamic), (c) participants attentional load (fast and slow Tetris speed), and (d) eccentricity of the targets from the primary display (35 and 45 of visual angle, or near and far). In addition to the four display combinations created by the two levels of color and motion, the fifth condition was a control display that contained no distractor elements. The experiment was divided into two 1-hr sessions. The first session included six blocks of Tetris, and the second session contained four blocks. Blocks lasted for 10 min, and 30 peripheral visual target signals (15 at each eccentricity) were presented in each block. Eccentricity was randomized within blocks. To increase uncertainty, the order of presentation of the five background conditions differed between high and low attentional load blocks. The five high-load blocks always preceded the low-load blocks to minimize the effects of practice as more blocks were completed. Overall, the sequence of background conditions was randomized across participants. The dependent measures in this study consisted of the detection rate for peripheral visual targets, the correct identification of the location of each target (near vs. far), and reaction times to targets. Procedure After reading and signing a consent form, participants were familiarized with the various display conditions and tasks. They were told that Tetris was their primary task, and that their performance would be measured in terms of the number of games they completed. Next, participants practiced Tetris until they reported feeling comfortable with performing the task and using the controls. The addition of a secondary task followed. During this dual-task training, participants were allowed to look over to the right-hand monitor where targets appeared before making a pos-

6 44 NIKOLIC, ORR, SARTER itive identification. Participants were instructed to press the left mouse button on detecting the left target and the right mouse button for the right target. They were told to respond as quickly as possible. Following the description of the peripheral target detection task, participants practiced Tetris once again. This time, their skill level was noted in an effort to equalize attentional load levels across participants with different Tetris-playing ability. Based on previous pilot studies, participants initially practiced Tetris at a difficulty level of 10. Depending on their performance at this level, they were then placed in one of two skill categories. If participants were able to carry on for 3 min without losing the game, they were placed in the high-skill category. All other participants were placed in the low-skill category. For the experimental trials, the Tetris speeds were adjusted across these two groups such that the high-skill participants were given faster speeds for the two attentional load levels than the low-skill participants. Once the participants felt familiar and comfortable with the experiment, the first block of Tetris was initiated. RESULTS Target Detection Rates For all targets that were detected, participants were able to correctly identify their location (either near or far). Furthermore, no differences were observed between participants placed in the high- and low-skill categories. A repeated measures analysis of variance (ANOVA) was performed on the target detection rates for the 300 signals presented to each participant. Due to inaccurate detection data for one participant, only 15 participants were included in the data analyses. As expected, the highest detection rate was observed in the control condition (M =.969, SD =.057), which was shown to be significantly higher than the next highest rate in the monochrome-static condition (M =.851, SD =.171), F(1,14) = 22.80, p <.001. The lowest target detection rate was observed in the color-dynamic condition (M =.649, SD =.282). There was a significant main effect for both color, F(1, 14) = 13.43, p <.01, and motion conditions, F(1, 14) = 35.98, p <.001 (see Figure 2), such that targets that were embedded in multicolor and dynamic displays were the most difficult to detect. The interaction between color and motion was marginally significant, F(1, 14) = 4.22, p =.059, suggesting that the combination of color and motion distractors had the most detrimental effect on detection performance. There was a main effect for target eccentricity (see Figure 3) such that participants detection rates were 20% higher for near targets than for far targets. Two participants failed to detect any far signals in the color-dynamic display condition.

7 ATTENTION CAPTURE AND DISPLAY CONTEXT 45 FIGURE 2 Mean target detection rates for the control condition and all color and motion display combinations. When color, motion, and control conditions were collapsed into five display levels, an interaction was observed between display and eccentricity, F(4, 56) = 4.793, p =.002, showing that far targets had the lowest detection rate in the color-dynamic condition (see Figure 4). No significant difference was found across attentional load (Tetris speed) levels. FIGURE 3 Mean target detection rates at each eccentricity level for the control and combined color conditions.

8 46 NIKOLIC, ORR, SARTER FIGURE 4 Mean target detection rates showing the interaction of display condition and target eccentricity. Reaction Times to Peripheral Visual Onsets Reaction times were recorded for all correctly identified targets. Due to incomplete reaction time data for 3 participants, only 13 participants were included in the following analyses. As with the detection data, there was a significant main effect for color, F(1, 12) = 4.77, p =.05, and motion, F(1, 12) = 15.58, p =.002 (see Figure 5). One-way ANOVAs revealed that only the dynamic display conditions had Mean reaction times for the control condition and all color and motion combina- FIGURE 5 tions.

9 ATTENTION CAPTURE AND DISPLAY CONTEXT 47 Mean reaction times at each eccentricity level for the control and both color con- FIGURE 6 ditions. significantly longer reaction times than the control condition, F(1, 14) = 8.98, p =.01. Participants in the color-dynamic condition were the slowest to respond to targets. Longer reaction times were observed for the monochrome-dynamic condition relative to the monochrome-static and the color-static conditions. An interaction effect was observed between distractor display color and target eccentricity, such that near targets were detected significantly slower in the color condition than in the monochrome condition, whereas far targets were detected equally slowly in both color conditions, F(1, 12) = 2.64, p =.012 (see Figure 6). When color, motion, and control conditions were collapsed into five display levels, an interaction was observed between display and eccentricity, F(4, 48) = 3.078, p <.05, such that in the monochrome condition, motion in the background (monochrome-dynamic) slowed reaction time for far targets only, whereas with a color display, both near and far targets were similarly affected by motion (color-dynamic; see Figure 7). An interaction was also observed between display and attentional load, F(4, 48) = 3.055, p <.05, such that participants in the color-dynamic condition were significantly slower to respond to targets when the attentional load of the primary task was higher (see Figure 8). DISCUSSION Designers in various domains tend to use the sudden appearance or flashing of a display element (e.g., the green outline box around flight mode annunciations on

10 48 NIKOLIC, ORR, SARTER Mean reaction times showing the interaction of display condition and target ec- FIGURE 7 centricity. the PFD on modern flight decks) to attract attention in a data-driven fashion. This design approach is widely used but not completely supported by recent findings in the attention literature, which indicate that visual onsets do not capture attention in a purely reflexive manner. Rather, the attention-capturing power of onsets can be modulated considerably by both top-down processes (e.g., the operator s attentional set) and bottom-up factors (e.g., other display elements and their behavior; Folk et al., 1992; Folk et al., 1998; Pashler et al., 2001). FIGURE 8 Mean reaction times showing the interaction of display conditions and primary task attentional load.

11 ATTENTION CAPTURE AND DISPLAY CONTEXT 49 This study focused on the latter aspect display context. In particular, it examined the effects of color similarity and display dynamics on the attention-capturing power of abrupt visual onsets. As expected, detection performance was near perfect and reaction times were the quickest in the control condition where targets appeared against a solid black background. In this case, targets did not compete with any other surrounding objects or events for the participant s attention. No effect of target eccentricity was observed. However, some targets at both eccentricities were missed even in the control condition. One possible explanation for this finding is a temporary functional narrowing of their peripheral visual field when participants focused their attention on the puzzle-completion task during particularly demanding moments. Thus, some degree of top-down modulation of attention may have played a role in this, and all other experimental conditions, leading to inattentional blindness in some cases and the failure of onsets to capture attention in a reflexive manner. The lowest target detection rate was observed in the color-dynamic condition, which was modeled after the multicolor displays containing moving elements in real-world environments, such as the PFD, engine indication and crew alert system (EICAS), or electronic centralized aircraft monitor (ECAM) display on modern flight decks. The particularly poor detection performance in this condition might be the result of masking by surrounding moving display elements, the color similarity between target and background, or a combination of both factors. An analysis of the detection performance in the other experimental conditions can help clarify this issue. Participants in the monochrome-static, monochrome-dynamic, and color-static conditions performed equally well and detected significantly more targets than the color-dynamic group. Note, however, that, especially for far targets, an increase in reaction time was observed in the monochrome-dynamic condition compared to the two static conditions. One possible explanation for this finding is that, in the two static conditions, targets were perceived during their onsets and were able to capture participants attention quickly, whereas in the monochrome-dynamic condition, the onset itself was masked by the movement of surrounding display elements. Participants in this group detected as many targets as those in the two static conditions but they did so with longer latencies after the initial onset. They recovered from their initial error of omission due to the color difference between target and surround that could still be observed in peripheral vision after the actual onset had occurred. This suggests that changes or movements of surrounding display elements are more powerful than color similarity in mediating capture by new objects (see Martin-Emerson & Kramer, 1995). Furthermore, the interaction that was observed for reaction time between color, motion, and eccentricity suggests that a dynamic or busy display produces masking of targets that are further in peripheral vision, whereas the combination of colored and dynamic display elements is able to mask both near and far targets and slow reaction times accordingly.

12 50 NIKOLIC, ORR, SARTER Detection performance was also affected by target eccentricity. Except in the control condition, in which no differences were observed, significantly fewer targets were detected when these targets were presented further away from the visual focus of the participant. The effect was most pronounced in the color-dynamic condition. Furthermore, the eccentricity effect for target detection was observed with equal magnitude independent of Tetris speed, which suggests that it is not the result of attentional narrowing due to increased primary task difficulty. Given the absence of an eccentricity effect in the control condition, one possible simple explanation may be the fact that, with increasing distance between participants visual focus and the location of target onset, the number of dynamically changing objects between the two points increased, thus amplifying the masking effect (Martin-Emerson & Kramer, 1995; Schons & Wickens, 1993). It is important to note, however, that a higher attentional load did produce significant reaction time costs for targets embedded in moving and color-similar distractors. Therefore, attentional narrowing in the strict sense was not observed, but rather an attentional delay in detecting targets at greater eccentricities. The observed effects of eccentricity have implications for the design of effective automation feedback. On modern flight decks, pilots are required to track changes and events that are embedded in complex displays, which are distributed widely across the cockpit. For example, the further away pilots attentional focus is from the flight mode annunciations on the PFD, the less likely they are to notice the appearance of the outline box indicating a mode change. For example, visual angles of 35 or greater from the PFD can result from a pilot looking out the window, down at the throttle quadrant, or at a central EICAS display. One possible solution to this problem was suggested by a recent study showing improved detection performance for visual onsets with a display that extended horizontally across the pilot s forward field of view (Nikolic & Sarter, 2001). Presenting a more spatially distributed signal accounts for the fact that pilots peripheral visual field moves with every fixation, and it minimizes the number of objects between attentional focus and target. Another possibility would be the placement of mode indications in a separate dedicated location, away from highly dynamic displays. Finally, the most powerful approach may be the distribution of information across various modalities, including touch (e.g., Sklar & Sarter, 1999), to address the scanning costs and intramodal interference caused by the increasing and almost exclusive reliance on visual information presentation (Sarter, 2000). In summary, the findings from this research help explain a subset of difficulties with pilot-automation coordination on modern flight decks why pilots miss the green box. At a more general level, they contribute to a better understanding of the effects of bottom-up processes on the attention capture power of visual onsets. They illustrate the need for careful adaptation and application of the most recent findings from the attention literature to the design of interfaces for complex systems and environments. Finally, our findings show the need for careful consider-

13 ATTENTION CAPTURE AND DISPLAY CONTEXT 51 ation of different levels of display design. Individual display elements (e.g., the flight mode annunciations and their respective boxes) might, in isolation, appear to be effective representations. However, they need to be evaluated in the context of surrounding display elements. Their density, visual properties, and behavior will affect the monitoring and interpretation of each individual element. Therefore, graphic components need to be considered in the context of graphic forms, process views, and, ultimately, the overall workspace (Woods, 1998). ACKNOWLEDGMENTS This work was funded, in part, by a grant from the National Science Foundation (Grant NSFII CAR; Technical Monitor: Dr. Ephraim Glinert and Dr. Gary Strong) and the Federal Aviation Administration (Grant DTFA 96-G-043; Technical Monitors: Dr. Tom McCloy and Dr. Eleana Edens). We would like to thank Sagar Reddy for his help with the software development and the students for their participation in this research. REFERENCES Abbott, K., Slotte, S., Stimson, D., Bollin, E., Hecht, S., Imrich, T., et al. (1996). The interface between flightcrews and modern flight deck systems. Washington, DC: Federal Aviation Administration. Chan, H. S., & Courtney, A. J. (1993). Effects of cognitive foveal load on a peripheral single-target detection task. Perceptual and Motor Skills, 77, Folk, C. L., Remington, R. W., & Johnston, J. C. (1992). Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human Perception and Performance, 18, Folk, C. L., Remington, R. W., & Wright, J. H. (1998). The structure of attentional control: Contingent attentional capture by apparent motion, abrupt onset, and color. Journal of Experimental Psychology: Human Perception and Performance, 20, Jonides, J. (1981). Voluntary versus automatic control over the mind s eye s movement. In J. B. Long & A. D. Baddeley (Eds.), Attention and performance IX (pp ). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Jonides, J., & Yantis, S. (1988). Uniqueness of abrupt visual onset in capturing attention. Perception and Psychophysics, 43, Martin-Emerson, R., & Kramer, A. F. (1995). Capture of attention by visual onsets. In Proceedings of the Human Factors and Ergonomics Society 39th annual meeting (pp ). Santa Monica, CA: Human Factors and Ergonomics Society. Nikolic, M. I., & Sarter, N. B. (2001). Peripheral visual feedback: A powerful means of supporting attention allocation and human-automation coordination in highly dynamic data-rich environments. Human Factors, 43, Pashler, H., Johnston, J. C., & Ruthruff, E. (2001). Attention and performance. Annual Review of Psychology, 52, Rensink, R. A., O Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8,

14 52 NIKOLIC, ORR, SARTER Rinalducci, E. J., Lassiter, D. L., MacArthur, M., Piersal, J., & Mitchell, L. K. (1989). Further experiments on the effects of foveal load on peripheral vision. Proceedings of the Human Factors Society 33rd annual meeting (pp ). Santa Monica, CA: Human Factors Society. Rinalducci, E. J., & Rose, P. N. (1986). The effects of foveal load on peripheral visual sensitivity. Proceedings of the Human Factors Society 30th annual meeting (pp ). Santa Monica, CA: Human Factors Society. Sarter, N. B. (2000). The need for multisensory feedback in support of effective attention allocation in highly dynamic event-driven environments: The case of cockpit automation. International Journal of Aviation Psychology, 10, Sarter, N. B., & Woods, D. D. (1997). Team play with a powerful and independent agent: Operational experiences and automation surprises on the Airbus A-320. Human Factors, 39, Sarter, N. B., & Woods, D. D. (2001). Teamplay with a powerful and independent agent: A full mission simulation study. Human Factors, 42, Schons, V., & Wickens, C. D. (1993). Visual separation and information access in aircraft display layout (Tech. Rep. No. ARL 93 7/NASA A3I 93 1). Savoy: University of Illinois, Aviation Research Lab. Simons, D. J. (2000). Current approaches to change blindness. Visual Cognition, 7(1 3), Sklar, A. E., & Sarter, N. B. (1999). Good vibrations: Tactile feedback in support of attention allocation and human-automation coordination in event-driven domains. Human Factors, 41, Theeuwes, J. (1991). Exogenous and endogenous control of attention: The effects of visual onsets and offsets. Perception and Psychophysics, 49, Williams, L. J. (1985). Tunnel vision induced by a foveal load manipulation. Human Factors, 27, Woods, D. D. (1998). Designs are hypotheses about how artifacts shape cognition and collaboration. Ergonomics, 41, Yantis, S. (1996). Attentional capture in vision. In A. F. Kramer, M. G. H. Coles, & G. D. Logan (Eds.), Converging operations in the study of visual selective attention (pp ). Washington, DC: American Psychological Association. Yantis, S., & Egeth, H. E. (1999). On the distinction between visual salience and stimulus-driven attentional capture. Journal of Experimental Psychology: Human Perception and Performance, 25, Yantis, S., & Hillstrom, A. P. (1994). Stimulus-driven attentional capture: Evidence from equiluminant visual objects. Journal of Experimental Psychology: Human Perception and Performance, 20, Manuscript first received March 2002