Integrating Psychophysiological Measures of Cognitive Workload and Eye Movements to Detect Strategy Shifts 1

Transcription

1 Integrating Psychophysiological Measures of Cognitive Workload and Eye Movements to Detect Strategy Shifts 1 Sandra P. Marshall C. W. Pleydell-Pearce B.T. Dickson San Diego State University University of Bristol QinetiQ, Ltd. smarshall@sciences.sdsu.edu c.pleydell-pearce@bristol.ac.uk btdickson@qinetiq.com Abstract This paper presents a case study to introduce a new technique for identifying and comparing cognitive strategies. The technique integrates eye movements with the Index of Cognitive Activity, a psychophysiological measurement of cognitive workload derived from changes in pupil dilation. The first part of the paper describes the technique and its constituent elements. The second part describes the task used in this study. The task has been used extensively in previous studies with electrophysiological recordings of EEG, EOG, and EMG and is know to elicit different levels of cognitive workload. The third part of the paper shows the results and provides detail about three separate strategies that emerged in the subject s performance. The strategies are first identified from differences in the Index of Cognitive Activity and corroborated through detailed analyses of the participant s eye movements as he performed the task. 1. Introduction As individuals learn to perform repetitive tasks, it is not unusual to observe improvements in their performance over time. In some cases, the improvement results merely from repetition keystrokes are learned, the eyes anticipate where to look next, and each step is taken just a bit more quickly. In other cases, improvement follows an abrupt change in strategy as the individuals gain insight into the task structure and modify their behavior. In reality, both causes for improvement are likely present for most tasks, but they are often difficult for observers to distinguish. 1 This research was supported by the Augmented Cognition Program, Defense Advanced Research Projects Agency (DARPA), under Program Officer Dylan Schmorrow. Strategy shifts are important because they reflect the cognitive effort that an individual gives to a task. A number of studies have documented that people tend to use strategies that demand lower cognitive effort when possible. For example, Gray and Boehm-Davis [1] noticed this phenomenon in their studies of a dual attention task developed for a collaborative MURI project to study cognitive workload. Not all strategy shifts are successful. Sometimes the new strategy results in poorer performance, and sometimes it requires more rather than less effort. However, two results should be evident when the new strategy is better than the old one. First, the individual performs at or above the previous level of performance, and second, the individual exerts less cognitive effort. This paper presents a case study to introduce a new technique for identifying and comparing cognitive strategies. The first part of the paper describes the technique and its constituent elements. The second part describes the task used in this study and its use in previous studies. The third part shows the results and provides detail about three separate strategies that emerged in the subject s performance. 2. The technique A promising technique for identifying and measuring shifts in cognitive strategies has emerged from an investigation of how cognitive workload changes over time on a task with known difficulty structure. The technique compares an individual s task performance against the Index of Cognitive Activity (ICA), a new method for evaluating cognitive workload from pupil dilation [2]. After potential strategy shifts are identified, the strategies themselves are analyzed in terms of the individual s eye movements to identify key components within them.

2 2.1. Eye movements The EyeLink II (SR International) was used in this study. The system provides binocular tracking at a sampling rate of 250 Hz. and records the horizontal and vertical pixel location for the point of gaze of each eye. For data collection in the present study, the participant wore a comfortable leather-padded headband having three miniature cameras. Two cameras with built-in illuminators track the eyes to provide accurate point-ofgaze data, and a third optical camera provides sufficient data to compensate for normal head movements. The newly developed integration of corneal reflections with pupil tracking algorithms permits very stable tracking of eye position by reducing the errors caused by headband slippage, muscle tremor, or environmental vibration. Eye movements were measured at all times during the participant's interaction with the computer display as he scanned the screen and performed the task. In addition, a videotape captured the participant s point-of-gaze superimposed on the image of the screen in real-time. effort, the ICA value will drop and remain lower than previous values. Ideally, the drop in ICA corresponds to a change in eye-movement patterns. A successful strategy shift would be accompanied by a more efficient scanning pattern and by lower values of the ICA. 3. The task The task used in the case study was the Gauge Task developed by Pleydell-Pearce and his colleagues at the University of Bristol [4-5]. The task was initially created for use with a number of electrophysiological recordings including EEG, EOG, and EMG. The case study reported here is the first use of the task with an eye-tracking system that provides both point-of-gaze and pupil dilation data Pupil dilation In addition to recording the point-of-gaze data, the EyeLink II also estimates the size of the pupil. These estimates taken 250 times per second yield a signal that can be processed to determine abrupt changes in pupil dilation. The Index of Cognitive Activity is a measurement based on the pupil dilation signal [2-3]. The Index is typically reported as the average number of abrupt discontinuities in the signal per second over a designated period of time. The Index can be computed for short time periods such as single trials in a multi-trial study as well as for long periods of time such as combat simulations. Some examples of its use can be found in [3] Strategy detection On repetitive tasks in which an individual is required to interact with a computer display, it is possible to trace graphically the history of the point of gaze for each repetition. If the individual is using a consistent strategy, each history trace will have approximately the same shape, with attention to specific features on the display emerging as concentrations of observation points. Thus, one way to identify strategy shift is to look for changing patterns of eye movements. A second way to identify strategy shift is to look for large changes in the measured Index of Cognitive Activity. If a shift has occurred, and if the shift corresponds to a strategy that requires lower cognitive Figure 1. The Gauge Task The Gauge Task as viewed by a subject is shown in Figure 1. Five gauges are displayed on the screen, with each gauge presented as a column of numerical data. The numerical values in each gauge oscillate continuously. The middle value for each one is zero, and the maxima of each gauge are ±50. Central pointers highlight the center of each gauge where readings are taken. When the gauge readings are positive numbers, the scale is shown in black numbers. When the gauge moves downwards into negative numbers, the scale is shown in red. Thus, each gauge is constantly changing, and the values shown for each one may be positive or negative. Below each gauge is a rectangular warning light that may be green, amber, or red, depending on the state of the gauge. The subject is required to monitor the five gauges and to keep each one from exceeding its boundary limits. When the gauge remains in the range of 0±15, the rectangular warning light located directly underneath the gauge is green. When the gauge exceeds 15 but is less than 20, the warning light turns amber. And, when the gauge exceeds 20, the warning light is red.

3 When the subject detects that a gauge has exceeded the safe boundary, he has the option of selecting that gauge for modification. To do so, he must use the arrow keys on the keyboard to toggle either left or right to the gauge to select the appropriate gauge. Then, he must use the arrow keys toggling up or down to change the polarity of the numbers, depending on whether he wishes the values in the gauge to increase or decrease. 1 When three gauges remain in a red state for more than 2 seconds within a single trial, the screen displays a simulated explosion and the trial ends. The task was designed with five levels of difficulty based on the number of gauges requiring attention at the same time. The lowest level of difficulty requires no gauge adjustment, i.e., all gauges remain within normal limits for the entire trial. The highest level requires nearly constant attention to maintain the gauges. Forty trials were created, with eight trials at each defined difficulty level. Each difficulty level appears once in every group of five trials, in random order. The trials are presented continuously, with a short break for the subject after each set of 10 trials. The total time for the task is approximately 40 minutes plus training time Task performance Task performance can be measured as a function of the number of gauges at any point in time that are amber or red, as the number of adjustments made by the subject during a trial, and by the number of blowups that occur. The number of gauges that are at an amber or red state has proven particularly useful, since it provides both direct information to the participant about current demand, and correlates very highly with actual level of intervention and negative-going DC potentials within EEG (Pleydell- Pearce, 1994; McCallum & Pleydell-Pearce, 1993). The best performance minimizes the number of times that any gauge exceeds the normal limits. The gauge data were recorded at a sampling rate of 5 Hz, yielding five measures per second across the entire 45- second trial, for a total of 225 epochs per trial. A gaugeby-trial (GT) value may be computed as the maximum number of 5 possibly green gauges minus the number of actual non-green gauges detected during the epoch. Two intervention scores may also computed: IV1 is 1 if the participant selected any gauge during the epoch and 0 otherwise. IV2 is 1 if the participant toggled a change in any gauge value, and 0 otherwise. These three epoch variables may finally be averaged over the 225 epochs for a single trial value for each of them. 1 In previous versions of the task, a joystick was used to change the direction of the gauge numbers Experimental details The case study was carried out as part of a larger collaborative study at QinetiQ, Ltd., Farnborough, England. The first author brought a portable eye tracking system to the QinetiQ facility for a series of tests as part of the DARPA Augmented Cognition Program. The participant was a QinetiQ employee who volunteered to take part. The entire study spanned several hours, with measurements taken from a number of different sensors including EEG, EOG, and EMG as well as eye tracking apparatus. The Gauge Task described here was one of several tasks used in the collaboration and was the final task presented to the subject. Prior to data collection, the Gauge Task was introduced by a member of the collaboration team who explained the task and allowed the subject to work through several example trials. Once the subject reached the necessary level of proficiency, the full 40-trial study was begun. Eye data were collected continuously during the full task and recorded for later analysis. 4. Results 4.1. Overall Performance The overall performance of the case study participant was excellent. He experienced only two blow ups, and both of those were relatively early in the task. Figure 2 shows the three performance variables across all trials. The two blow ups occurred on Trials 7 and 13. Figure 2. Three performance variables plotted across trials The performance variable GT was highly correlated with difficulty level, with R 2 =.87. Figure 3 shows the distribution of GT at each level of difficulty, with eight trials shown at each level. The two intervention measures are almost identical and both correlate negatively with GT (R 2 =.87) and positively with level (R 2 =.96). These high values confirm that the structure of the task into the five levels of difficulty did result in five increasingly difficult levels of performance as designed.

4 Second, the levels were increasingly more effortful as difficulty rose, as shown in Figure 5, with level 5 trials resulting in substantially higher values of ICA. The differences among levels was statistically significant (F=2.78, df=4,35, p<.05), and the linear trend across the five levels is significant as well (F=11.03, df=1,35, p<.01) Strategy shifts Figure 3. Distribution of GT across difficulty levels 4.2. Index of Cognitive Activity The Index of Cognitive Activity was computed for each second of each trial, and a mean value for each trial was also obtained. The mean results are shown in Figures 4 and 5. Figure 4 shows the normalized index across the trials in the order that the trials were attempted. Figure 5 shows the same normalized index with the trials grouped by level of difficulty. Figure 4. Mean values of ICA over trials These two figures provide important information. First, the participant clearly improved over time in his performance, with a dramatic shift occurring approximately half way through the trials, as shown in Figure 4. Figure 5. Mean values of ICA by difficulty levels To illustrate the important changes that were observed in strategy, this paper focuses on three trials, each of level 5 difficulty: Trial 3, Trial 16, and Trial 38. The three trials each began with seconds of all green gauges, and the participant made no response during the first 15 seconds of each one. Thus, the analyses below are based on the remaining 30 seconds of each trial. These trials capture performance very early in the task, approximately midway, and at the end of the task. The three trials yielded very different ICA values, as shown in the above figures, with Trial 3 at a very high level, Trial 16 at a moderately high level, and Trial 38 at a low level. The drop in ICA from Trial 3 to Trial 16 to Trial 38 is certainly due in part to the repetitive nature of the task after 38 trials, the participant was responding more quickly. However, different strategies are also apparent from analysis of the eye movements. The total eye movements for the three trials are shown in Figure 6. For parsimony, the movements of the left eye only are depicted. In all three cases, the point- of-gaze for the right and left eyes are essentially the same. The patterns in these three figures are quite different. In Trial 3, the eye is drawn primarily to the numbers in the gauges that are rapidly changing. Relatively little time is spent attending to the warning signals underneath the gauges. Frequently, the eye goes to the central region defined by arrows in the center of the gauge. In Trial 16, the eye is drawn first to the warning signals and then to the numerical values of the gauges. Moreover, when the point of gaze shifts to the numbers, it moves only to the lowest numbers in the gauges, not to the central highlighted region. This pattern contrasts with the movements of Trial 3, where the entire numerical range of the gauges received attention. Finally, in Trial 38, the point of gaze rarely leaves the warning signals and the bottom-most numerical value of each gauge. Virtually no attention is given to the numerical values in the middle or upper portion of the gauges. Statistical tests confirm the differences among the patterns of eye movements. Each trial can be analyzed according to the number of times that five specific eye movements occur: scan only one numerical value within a gauge, move from one numerical value to another within the same gauge, move from a numerical gauge to

5 warning light, move from a warning light to a numerical gauge, or scan the warning lights only. Figure 7 gives the distribution obtained from this analysis. A chi square test of proportions confirms that these distributions are significantly different (χ2=45.26, df=8, p<.01). Trial 3 Figure 7. Distributions of eye movements on three trials Thus, the eye movements reveal three distinct patterns as the participant engaged in the task. How do these patterns relate to different cognitive strategies? Part of the answer comes from the initial introduction of the task to the participant. When the participant received instruction about the task, he was directed to monitor the numbers in the gauges in order to keep them in the normal range. Obediently, he began the task by looking at the numbers as they changed value. His strategy, as deduced from the eye movements, was to scan across the gauges themselves, to evaluate the size of the numbers, and to make an appropriate response to adjust the gauge when he judged the numbers were too high. This strategy requires a great deal of cognitive effort. The participant first had to make the decision to move his eyes to a particular gauge and to process the size of the numbers in that gauge. Next, he must judge whether the numbers were out of range. Finally, he had to make the physical response of toggling the arrow keys to bring the numbers back into acceptable range. Not only is this strategy effortful, it is time consuming. Often the participant looked at several numbers within the same gauge, moving his eyes up and down the values, apparently to confirm that a gauge was indeed exceeding its normal boundaries. By Trial 16, the participant has developed a better strategy. At this point, he does not look initially to the gauges. Instead, he pays attention to the warning signals. His eye movements show that he looks across the signals (not across the gauges themselves) and the movement is more often from signal to numbers than from numbers to signal. He still expends significant cognitive effort to perform the task. The initial step now is to recognize a red or amber signal as the initial gauge selector. Next, he must move his eyes upward and process whether the needed adjustment is positive or negative. And, as before, he must then make the physical toggle response. Trial 16 Trial 38 Figure 6. Patterns of eye movements on three trials With this strategy, he has rearranged the order in which he performs the necessary comparisons, and the results are a quicker response and somewhat less cognitive effort. It is unclear by the eye movements alone whether the participant is attending to the numerical

6 values in the gauges or to the color of these numerical values (negative values are red, positive values are black). Finally, by Trial 38, the participant has moved to what may be a nearly optimal strategy. The most striking feature of this strategy, is the almost total absence of directed eye movements to the gauges themselves. The movements now are horizontal rather than vertical, and the point of gaze stays almost totally on the warning signals or on the bottom-most numerical value of the gauges. The participant appears to have recognized that color alone is sufficient for adjusting a gauge up or down, because he seems to be relying on peripheral vision now to determine his physical response. This strategy is efficient. It results in quicker responses, as shown by the fact that his responses were so quick on this trial that very few gauges moved from amber to red. Only rarely did more than a single gauge stay in the warning zone for longer than one or two measurements (i.e., for more than.2-.4 seconds). The strategy also apparently requires less cognitive processing, because it relies on color detection in the peripheral area instead of text processing of the numerical data. Moreover, this strategy does not require that the participant deliberately move his eyes to a specific point to confirm a warning signal. It should be noted that in a post-experiment interview, the participant mentioned that after several trials, he figured out a way to do the task. He was not able to articulate fully how his strategy changed or what caused the new strategy to develop. Nonetheless, he was aware that the strategy shift made the task easier to perform. 5. Summary The case study described here demonstrates that strategy shifts may be identifiable from eye data. First, the changes in pupil dilation, as measured by the Index of Cognitive Activity, identify the timing and location of potential strategy shifts. Abrupt decreases in the computed index suggest that the individual has changed the way that he is performing the task. Second, after the shifts have been identified, the individual s performance before and after a shift can be examined more closely. This second analysis focuses on the participant s point of gaze as he carries out the task. For repetitive tasks such as the one used here, the resulting patterns can be very revealing. Although this study involved a single subject and does not allow comparison among individuals, it is interesting to note that the results found here correspond generally to those found from EEG studies of the same task. Pleydell- Pearce (1994) and Pleydell-Pearce et al (1995) found strong correlations between cognitive workload as measured by task difficulty and scalp-surface negativity. The Index of Cognitive Activity also correlates highly with task difficulty. Thus, shifts in cognitive effort and attention are evident in at least two physiological measurements. The similarity between the findings from EEG data and from pupil dilation data is important for at least two reasons. First, the EEG data are derived from explicit scalp locations and are intended to monitor activity in specific cortical areas. EEG studies are hard to conduct and labor intensive in analysis. If the ICA were used as a precursor to EEG investigations, researchers might be able to target their EEG probes more efficiently. Second, the very detailed EEG data and results can also direct use of the Index of Cognitive Activity. EEG research is by nature a laboratory experiment. In contrast, the portability of eye tracking apparatus allows the ICA to be measured in the field. In future research, EEG studies may first indicate activity in various cortical regions in detailed laboratory studies, and the more general Index of Cognitive Activity may subsequently be used to validate the same phenomenon in field study. Together, these two approaches bring researchers closer to understanding the nature of cognitive strategies. 6. References [1] W. Gray, & D. Boehm-Davis, Presentation at the Formal Review of the Multidiscipline University Research Initiative (MURI) Project, Understanding and Measuring Cognitive Workload: A Coordinated Multidisciplinary Approach, awarded to George Mason University and San Diego State University, Grant F , Fairfax, VA, November [2] S. Marshall, S., Method and Apparatus for Eye Tracking and Monitoring Pupil Dilation to Evaluate Cognitive Activity, U.S. Patent 6,090,051, July [3] S. Marshall, S. The Index of Cognitive Activity: Measuring Cognitive Workload., In D. Schmorrow (Chair), Tomorrow s Human Computer Interaction from Vision to Reality: Building cognitively aware computational systems. Symposium presented at IEEE 7 th Conference on Human Factors and Power Plants, Scottsdale, AZ, September [4] C. W. Pleydell-Pearce, DC Potential Correlates of Attention and Cognitive Load, Cognitive Neuropsychology, 11, 1994, [5] C. W. Pleydell-Pearce, W. C. McCallum, & S. H. Curry, DC shifts and cognitive load. In Karmos, M. Molnar, V. Csepe, I. Czigler & J. E. Desmedt (Eds.). Perspectives of Event Related Potentials Research. Supplement 44 to Electroencephalography and Clinical Neurophysiology (pp ). Elsevier, Amsterdam, 1995.