Using Task-induced Pupil Diameter and Blink Rate to Infer Cognitive Load

Transcription

1 Using Task-induced Pupil Diameter and Blink Rate to Infer Cognitive Load Running Head: Using Task-induced Pupil Diameter and Blink Rate to Infer Cognitive Load ABSTRACT Minimizing user cognitive load is suggested as an integral part of human-centered design, where a more intuitive, easy to learn and adaptive interface is desired. In this context, it is difficult to develop optimal strategies to improve the design without first knowing how user cognitive load fluctuates during interaction. In this study, we investigate how cognitive load measurement is affected by different task types from the perspective of the Load Theory of Attention, using pupil diameter and blink measures. We induced five levels of cognitive load during low and high perceptual load tasks and found that although pupil diameter showed significant effects on cognitive load when the perceptual load was low, neither blink rate nor pupil diameter showed significant effects on cognitive load when the perceptual load was high. The results indicate that pupil diameter can index cognitive load only in the situation of low perceptual load, and are the first to provide empirical support for the cognitive control aspect of the Load Theory of Attention, in the context of cognitive load measurement. Meanwhile blink is a better indicator of perceptual load than cognitive load. This study also implies that perceptual load should be considered in cognitive load measurement using pupil diameter and blink measures. Automatic detection of the type and level of load in this manner helps pave the way for better reasoning about user internal processes for human-centered interface design. 1

2 CONTENTS 1. INTRODUCTION 2. RELATED WORK 2.1. Cognitive load measurement 2.2. Separating perceptual load and cognitive load 3. HYPOTHESES 4. METHODS 4.1. Tasks and procedure 4.2. Participants 4.3. Feature extraction 4.4. Analysis 5. RESULTS 5.1. Subjective rating of mental effort 5.2. Performance score 5.3. Blink rate 5.4. Pupil diameter 6. DISCUSSION 6.1. Results discussion 6.2. Limitations of the experiment 6.3. Implications for HCI 2

3 1. INTRODUCTION The purpose of this paper is to separate perceptual load and cognitive load as two distinct sources of task difficulty rather than characterizing cognitive load as the only one, and to analyze pupil diameter and blink rate as responses to the two types of load, with an aim to diagnose the type and level of load in interface evaluation and in interactive systems to allow the system to proactively adjust to the user s needs and demands. The above issue is important since cognitive load measurement is attracting growing interest in human computer interaction (HCI) studies as we move from the principle of users can adapt to the era of humancentered design (Oviatt, 2006). Emerging research related to the user s cognitive load includes distributed cognition in interaction, where off-loading cognitive effort and improving cognitive load-balancing are part of the core principles (Hollan, Hutchins & Kirsh, 2000; Hollender et al., 2010), computer-based mediators, where cognitive load measurement is required for dynamic task allocation (McFarlance, 2002) and for reducing mediation cost (Iqbal et al., 2005). Usability testing is another area in which the concept of cognitive load is applicable (Hollender et al., 2010). In addition, cognitive overload detection is important to maintain user mental health during sustained interaction (Cinaz et al., 2010). As such requirements grow, quantifying user cognitive load becomes an essential step to progress these research areas. Exploring pupil diameter and blink rate to index cognitive load has been an active research line in several application domains. For example, pupil diameter and/or blink rate have been employed in flight tasks (Wilson, 2002; Veltman & Gaillard, 1998; Jessee, 2010), combat management (Van Orden, et al., 2000; de Greef et al., 2009), and air traffic control (Brookings, Wilson & Swain, 1996) to monitor operators cognitive load in critical missions. They were also studied for task analysis to improve human computer interaction (Iqbal et al., 2005; Haapalainen et al., 2010; Chen, Epps & Chen, 2013a) and for constructing effective and user-personalized training (Chen et al. 2011a; Sibley, Coyne & Baldwin, 2011). Furthermore, pupil diameter and/or blink measures have been found useful to the system usability (Nakayama & 3

4 Katsukura, 2007; Józsa 2011) and can differentiate the load changes in visual and audio presentation types (Klingner, Tversky & Hanrahan, 2011), ages (Piquado, Isaacowitz & Wingfield, 2010), intelligence (Beatty, 1982), and experts and novices (Richstone et al., 2010). A variety of applications related to cognitive load make this research on eye activity an exciting and significant area. However, existing work on cognitive load measurement has focused on just a single task type at a time, and our understanding of how cognitive load measurement varies from one task type and level to the next is incomplete. Requirements of differentiating ordinal cognitive load levels reliably, instead of, for example, two discrete states of thinking and resting (e.g. Marshall, 2007), also make it a challenging research field. Our focus on separating perceptual load in cognitive load measurement was motivated by the different sources of load production described in Cognitive Load Theory and Multiple Resource Theory, and particularly by the relationship of perceptual load and cognitive load described in the Load Theory of Attention. The definition of cognitive load can be found in Cognitive Load Theory: a multidimensional construct representing the load that performing a particular task imposes on the learner s cognitive system (Paas et al., 2003). It provides theoretical framework to reduce extraneous load, e.g. associated with split-attention effect, modality effect and redundancy effect (Hollender et al., 2010). Cognitive Load Theory emphasizes the element interactivity causing the load on limited working memory, as some elements need to be held while others are being processed concurrently in working memory (Sweller, 2010). The element is anything that needs to be learned during the task, and the number of interacting elements determines the element interactivity levels (Sweller, 2010). However, the focus of how load produced is different to some extent with the well-known Multiple Resource Theory, which highlights the multiple resources and their difficulty in timesharing that produce the load (Wickens, 1991; Wickens, 2008). Any task can be represented by one or all dimensions of the 4

5 resources in different levels, that is, visual and auditory in the modality dimension, spatial and verbal in the coding dimension, perceptual/cognitive and action/response in the stage dimension, and focal and ambient in the visual channel dimension. The term mental workload is often used in these studies of load estimation with Multiple Resource Theory. With regard to inducing load and measuring it, the two terms, mental workload and cognitive load, are interchangeable, but in this work we prefer cognitive load as we varied the interacting elements in working memory to produce the task demands, for example, in contrast to adopting multiple tasks where multiple resources were employed to exhaust user mental capacity in previous studies. As a relevant process to cognition, perceptual load is another important factor in human information processing. Perceptual load is described in the Load Theory of Attention as the requirement to perceive more items, particularly in a short period of time (Lavie, Hirst & de Fockert, 2004). For example, increasing the number of items in a search task can produce a higher perceptual load. To resolve the longstanding debate of when irrelevant information perceived, that is, in early perceptual processing or later postperceptual process (memory or response selection), the Load Theory of Attention clarified that cognitive control, which depends on working memory, typically occurs in situations of low perceptual load to maintain processing priorities for task goals. High perceptual load can exhaust processing capacity, leaving no capacity available for distractor processing (Lavie, Hirst & de Fockert, 2004). It is worth noting that some work also used the term of visual load to describe the visual demanding (e.g. Engstrom, Johansson & Ostlund, 2005). However, perceptual load is not explicitly mentioned in Cognitive Load Theory as a separate factor to the cognitive load occurring in working memory. Although perception and cognition are pointed out as two different stages in Multiple Resource Theory, the main concern is whether the total demands from them exceed the limited mental capacity. How the two types of load, cognitive and perceptual, actually interact with each other, and to what extent perceptual load affects the reliability and sensitivity in cognitive load measurement, have not been addressed. With new theories and models of cognition and perception 5

6 emerging, this invites empirical work to form a better understanding of their relationship and to find avenues for more accurate measurement. In this study, we examine the extent to which the Load Theory of Attention can explain observed perceptual and cognitive load effects, based on pupillary response and blink rate. Our contributions are as follows: (1) We demonstrate the effects of both perceptual load and cognitive load manifested in pupil diameter and blink rate, extending the use of these two measures for only cognitive load measurement in previous studies. (2) We confirm that the responses of pupil diameter and blink rate to cognitive load fit the Load Theory of Attention in terms of the relationship between perceptual load and cognitive load, and interpret cognitive load with considerations of perceptual load level. (3) We reveal the different affordances of pupil diameter and blink rate for perceptual load and cognitive load measurement. 2. RELATED WORK 2.1 Cognitive load measurement There is a general agreement that cognitive load is due to limited working memory, and is produced during the interaction between task demands and users with different characteristics. Cognitive load can be seen as an effect that users experience, while mental effort is a unit that users actually exert in response to the load (Jong, 2009). The actual exerted responses can often be implied from analytical and empirical methods in general. NASA Task Load Index (TLX) is a widely used subjective rating measure. With this measure, users are required to rate six subscales of workload, and an overall workload can be calculated by the sum of each weighted subscale (Hart & Staveland, 1988). Another simple and popular measure is the subjective cognitive load measure (SCL), which uses only a single scale to report the overall experienced mental effort in task. It has a 7- or 9-point symmetrical category scale, and the rated mental effort is then translated into a numerical value (Paas, 1992). SCL has been reported to be quite reliable based on numeric indicators (Paas et al., 2003) and has similar performance with weighted version of NASA-TLX (Wiebe, Roberts & 6

7 Behrend, 2010). Even though, for realistic situations, this approach seems impractical, because the rating questionnaires not only interrupt the flow of tasks but also add more tasks for potentially overloaded users. Another technique is performance measurement, often including reaction time and error rate on either primary or secondary tasks. Previous studies yielded interesting findings on the relationship between performance, task demands and mental effort. In the regions of very low and very high demands of tasks, performance is not sensitive to the demands anymore, because performance can be maintained through either increased mental effort when tasks become demanding or decreased mental effort when tasks are too demanding to continue (Eggemeier & Wilson, 1991). Likewise, it seems that performance deterioration cannot easily be attributed to one or the other of increased perceptual load or cognitive load, therefore the strategies to reduce cognitive load cannot be easily formalized. For example, if it is because of high perceptual load, we can reduce the density of information displayed while if it is due to high cognitive load, maybe providing a sketchpad to store interim information is more useful. Besides these techniques, studies have explored objective methods based on psychophysiological measures. Pupillary response and blink are two examples. Early research achieved fruitful results, suggesting blink and, especially, pupillary response as indices of mental workload in different tasks (e.g. Veltman & Gaillard, 1998; Brookings, Wilson & Swain, 1996; Van Orden et al., 2001; Beatty, 1982; Stern, 1984; Kramer, 1991). However, pupillary response has gained more popularity than blink because it is sensitive to a wide range of processing activities, e.g. perceptual, cognitive and response related demands, therefore it is not diagnostic of the type of these activities (Kramer, 1991). Although blink appears to be sensitive to cognitive load, its trend is often inconsistent between different studies. For example, studies using search tasks have often found that blink rate decreases when there is a requirement to search more objects (Van Orden et al., 2001; Chen et al., 2011a; Irwin & Thomas, 2010). Meanwhile, it has been reported that blink rate increases in arithmetic tasks (Veltman & Gaillard, 1998; Recarte et al., 2008; Chen, Epps & Chen, 2011b), and in conversations and mental rehearsals (Stern, 1984). 7

8 All the links with cognition demonstrated by pupillary response and blink imply that the two measures are capable of indicating cognitive load, but they also have their own limitations as an indicator, which might be worth investigating for an accurate measurement. 2.2 Separating perceptual load and cognitive load According to the Load Theory of Attention (Lavie, Hirst & de Fockert, 2004), there are two dissociable mechanisms: a rather passive perceptual selection mechanism in the early perceptual process, where information can be noticed or unnoticed, and a more active cognitive control mechanism in the late postperceptual process, where working memory is involved. The necessity to separate the two types of load can be supported by their opposite influences on distractors in terms of interference. That is, distractor interference decreases when perceptual load is high while the interference increases when cognitive load is high (Lavie, Hirst & de Fockert, 2004). In this study, our aim is not to separate perception and cognition in process at each moment, but to show a distinction between the two types of load that cause task difficulty during a period. From this perspective, the means by which task demands were modulated in previous studies can be broadly characterized as (i) using dual tasks or multiple tasks involving different resources that cannot be easily shared simultaneously (e.g. Cinaz et al., 2010; Wilson & Russell, 2003; Veltman & Gaillard, 1998; Marshall, 2007), (ii) increasing the number of events needing to be handled at the same time, e.g. using visual tasks (e.g. Brookings, Wilson & Swain, 1996; Van Orden et al., 2001; Haapalainen et al., 2010; Chen at al., 2011a), (iii) increasing the interactivity of items needing to be processed in working memory, e.g. using arithmetic tasks or other tasks without changing the number of task stimuli at each time, (Iqbal et al., 2005; Cinaz et al., 2010; Haapalainen et al., 2010; Chen, Epps & Chen, 2011b; Klingner, Tversky & Hanrahan, 2011; Piquado, Isaacowitz & Wingfield, 2010). The main focus is often on whether cognitive load has been properly induced, indicated by poorer task performance or higher subjective rating, or on the effectiveness of psychophysiological measures, and the 8

9 type of load is often ignored. When target density (Van Orden et al., 2001), positions of basketball players (Chen et al., 2011a) and traffic volume (Brookings, Wilson & Swain, 1996) were varied from low to high to induce low and high cognitive load (according to (ii)), the perceptual load was also implicitly changed from low to high, a confounder not acknowledged in their analyses. In this study, we separate task demand modulation strategies (which we term task types ) (ii) and (iii) and take them as methods for inducing perceptual load and cognitive load respectively, based on their different means for manipulating the demands on working memory and the number of task stimuli required to be perceived, to investigate their influence. A key consideration for cognitive load measurement is that perceptual load can change the amount or attribute of information processed in working memory. For example, an empirical study found that visual and cognitive load have different effects on driving performance (Engstrom, Johansson & Ostlund, 2005). However, this insight has yet to be fully accounted for in cognitive load measurement research. To what extent the level of perceptual load influences the effectiveness of subjective rating, performance score and eye-based measures has not been elucidated. One recent study (Jessee, 2010) directly suggested separating visual workload (we use the term of perceptual load following the terminology of the Load Theory of Attention) from the broad workload domain for a more specific and clear understanding of the nature of systems and design implications. The author attempted to discriminate visual and mental workload by task difference, pilot on-controls and pilot off-controls, while holding the task difficulty constant (indicated by no significant difference in workload scale ratings), and used hovering flight and actions on contact as the low and high mental workload respectively (indicated by the significant difference in workload scale ratings). Shorter blink interval (higher blink rate) was found in pilot off-controls (low visual demands) and no statistical difference in pupil diameter was found in any cases. Although blink interval was suggested as being able to discriminate task difference in visual demands in this study, the validation of manipulating visual workload was hardly demonstrated (only workload rating 9

10 was used to show no difference in task difficulty) and using different tasks to be the low and high mental workload cannot rule out other confounding factors from visual demands because no explicit task stimuli were controlled. Furthermore, whether it was the low perceptual load that caused higher blink rate and how these two different workloads affecting each other was not explained. Another study (Recarte et al., 2008) explicitly included and excluded visual demands concurrently with cognitive tasks to investigate the validity of NASA-TLX, pupil diameter and blink rate measures of mental workload. In their study, the three measures were analyzed in three single cognitive tasks (listening, talking and calculating), without any visual demands involved, and in three dual tasks (detecting and listening, detecting and talking, detecting and calculating) with visual search required. Based on NASA-TLX, they categorized talking and calculating as high mental workload tasks while detecting and listening as low mental workload tasks. They found that blink rate was higher in higher load tasks but it decreased with the inclusion of visual demands in dual tasks. Therefore, blink rate can differentiate between visual and mental workload, but NASA-TLX and pupil diameter were not able to do it. This study clarifies how visual demands directly influence blink rate by including and excluding visual demands involved in tasks. However, inducing visual demands to the single tasks as the dual tasks imposed higher cognitive load on participants, according to Multiple Resource Theory, confirmed by larger pupil diameter in dual tasks than in single tasks in this study. From this perspective, the low blink rate in dual tasks and high blink rate in single tasks still cannot exclude the possibility of being due to cognitive load change by multiple resources (modalities). Furthermore, the type of source causing the difference in pupil diameter change in single and dual tasks was not well explained by the authors. Therefore, in this study, we manipulate low and high perceptual load levels in two types of visual tasks, according to the number of items required to be perceived, rather than in two extreme conditions, with and without visual demands. Under each perceptual load level (single resource), we manipulate five levels of cognitive load, based on the number of items required to be hold in working memory, verified by subjective rating and performance 10

11 measures, to find out how pupil diameter and blink rate change with cognitive load, rather than just using subjective rating measure to group two cognitive load levels from different tasks. 3. HYPOTHESES Based on the Load Theory of Attention, under situations of high perceptual load, not all stimuli are perceived due to limited perceptual capacity. However, when the perceptual load is low, all stimuli can be perceived, and working memory is required to maintain the current processing priority, so that high priority targets can compete for attention with low priority distractors to guide the behavior in accordance with current goals. The effects of perceptual load and cognitive load on selective attention are additive, independent and opposite on distractor interference (Lavie, Hirst & de Fockert, 2004). Therefore, if participants are overloaded in the perception stage, varying the demands on working memory is unlikely to invoke more mental effort, and might result in similar rates of mental effort and task performance. On the contrary, if the perceptual load does not disturb the perception of stimuli, the invested mental effort is more likely to be varied with the demands on working memory. We thus posit: Hypothesis 1: Tasks with high perceptual load will result in little change in subjective rating of mental effort and in task performance in spite of varying levels of induced cognitive load. This insensitivity to cognitive load is not expected for tasks with low perceptual load. The expected trend of blink rate under each perceptual load condition is similar to that in hypothesis 1. Furthermore, since high blink rate indicates few items and low blink rate indicates a large number of items needing to be processed in a single type of task in previous studies, if perceptual load is the dominating factor, the same blink pattern is expected to be observed when the number of items varies between different types of tasks. We thus posit: Hypothesis 2: Blink rate increases with cognitive load during low perceptual load tasks but changes little during high perceptual load tasks. Blink rate is higher during low perceptual load tasks than during high perceptual load tasks, regardless of the level of cognitive load. 11

12 Studies of task-invoked pupillary responses for decades have confirmed that pupil diameter reflects mental effort within task and between tasks (Beatty, 1982). As both perceptual load and cognitive load can make tasks challenging, participants need to invest mental effort to deal with either type of load in order to achieve the task goals. We thus posit: Hypothesis 3: Pupil diameter increases with cognitive load level during low perceptual load tasks, but changes little during high perceptual load tasks. Pupil diameter variation is high during high perceptual load tasks, and in the high levels of cognitive load during low perceptual load tasks. 4. METHODS We conducted an experiment to assess the capability and limitations of pupil diameter and blink to index the cognitive load in low and high perceptual load tasks. The trend of blink rate and magnitude of pupil diameter change was examined in both conditions. The consistency of pupil diameter change and blink rate with cognitive load level in each condition was compared, to understand how perceptual load affects cognitive load measurement. 4.1 Tasks and procedure The low perceptual load task was a mental arithmetic task (Figure 1, top row). Participants were required to add four numbers sequentially displayed in 3-second intervals on the screen. After 12 s, they used the mouse to select the correct answer from 10 displayed choices. Task difficulty levels were regulated by the number of digits for addition and carries produced by addition. In the lowest cognitive load level, one-digit addition was required but no carry was produced, while in the highest cognitive load level, three-digit (within 200) addition was carried out with two carries generated during each addition process. Specifically, the four integers were randomly selected from one of the following groups: (i){0,1}, (ii){5 to 9}, (iii){10 to 19}, (iv){84 to 93}, (v){175 to 179, 185 to 189} for one of the five levels of cognitive load. 12

13 The high perceptual load task was a search task (Figure 1, bottom row), in which participants were required to find four target digit strings in 12 s. In the lowest cognitive load level, the target was a one-digit number. We incremented one digit for each cognitive load level, therefore in the highest cognitive load level, the target was a five-digit number. The non-target numbers were made up of the same digits but in different sequences, to prevent the participant from memorizing only a few digits. In each cognitive load level, four targets appeared in each of the four paths at different times and every number moved in their respective paths every 3 s by one box respectively, for 12 s in total (up, down, left, right for the four paths, randomly chosen in each trial). Participants were instructed to click as many targets as possible, without knowing the digits of targets. When correctly clicked, the target changed to zeros, and the number of zero digits depended on the cognitive load level. Figure 1. Time line of the arithmetic task (top) and the search task (bottom). Before the commencement of each task, there was a 2 s adjustment period, for the eye to adapt to the background. The time window of interest is 12 s, indicated by the beginning and ending of task stimuli. 13

14 The perceptual load was controlled by the number of objects that participants needed to perceive in order to achieve the task goal. In the low perceptual load (arithmetic) task, for every 3 s, participants needed to perceive 10 objects, 9 X s (distractors) and 1 addend (target). The target and distractors can be easily identified by peripheral vision, and up to 3 digits were required to be perceived by foveal vision. However, in the task with high perceptual load (search), for every 3s, participants needed to perceive 28 numbers to find 4 targets, but the targets and distractors were similar in appearance, therefore, foveal vision was needed for each digit perception. Under each perceptual load condition, during 12 s, cognitive load was manipulated by the number of digits of the sum and carry in working memory in the arithmetic task, and the number of digits of target strings in the search task since the digits perceived were not necessarily to be held all in working memory. Tasks with the lowest and highest cognitive load level were designed to be easiest and difficult enough that participants could neither work them out easily nor give up easily. The intermediate load levels between were designed to progress uniformly. From this design perspective, the load level ranges reflect the minimum and maximum cognitive capacity under low and high perceptual load levels. Participants completed the tasks facing a PC running Windows 7, with a screen resolution of They were required to wear the frame of a pair of safety glasses, throughout four segments of tasks each day. Two lightweight infrared web cameras were mounted on the glasses frame, one pointing to each eye to track the pupil. The two cameras were connected to two laptops through USB 2.0 and the eye behavior was recorded at 30 Hz in AVI format. On each day, the participants completed four task segments, none of which lasted for more than 25 minutes. At the beginning of each task segment, there was a synchronization procedure so that the task presentation timestamp recorded in the PC was aligned with the video timestamp. We started with a baseline procedure. Participants were asked to look at count-down numbers which were displayed sequentially with an interval of 1.5 s in the center of the screen, without any task goal for 4 s. 14

15 Then a continue button appeared for the participant to click through to the next step. This procedure was conducted six times with backgrounds changing between white, grey, black then white, black and grey. The baseline procedure on each day, 24 s in total, enabled us to estimate how quickly the pupil adapted to the background change, and the average blink rate in non-task state. After the baseline procedure, tasks of different type and level were performed. In the first segment on each day, two repetitions were performed for each task type and level. At the end of each task, there was a rating form for mental effort using SCL, selected by participants and automatically recorded by the presentation program. The categories corresponding to the nine points ranged from very little to near maximum. A performance score was also automatically recorded as 1 if participants selected the correct arithmetic answer or clicked the target, otherwise 0. The other three task segments followed the same procedure but there was no rating form at the end of each task. This was to avoid annoyance by frequently disturbing the participants, who had many tasks to perform. The participants continued to execute tasks until a sign showing the end of the task segment. The sequence of tasks comprised permutations of the task type and level in random order, which were then divided into two parts for day 1 and day 2. Half of the participants began with the day 1 tasks and half began with the day 2 tasks. Each participant had a break between task segments. 4.2 Participants 22 volunteers (9 female and 13 male, age: M=26.8, SD=4.0), successfully completed the tasks in two different days. A short training period was provided on the first day, before the experiment. This study had ethics approval and each participant signed a consent form and received two vouchers as compensation for their time. 4.3 Feature extraction Blinks and pupil diameter were extracted from video recordings using scripts developed in MATLAB (Chen & Epps, 2014). Estimated pupil curve were superimposed on the video for visual checking to 15

16 identify obviously incorrect pupil diameters and falsely detected blinks. Blink was defined as the pupil being occluded by at least half. Manual corrections of the small proportion of falsely detected blinks (1.2%) were made. The pupil diameter measurement was based on the length (in pixels) of the major axis of a fitted ellipse. The pupil dilation signal (as a function of time) was linearly interpolated during blinks (where the pupil diameter strictly equals zero) and then was filtered by a median filter of length 3 frames, to remove noise typically caused by rapid eye movements. The final pupil diameter was converted into a millimeter scale, using measurements of the actual eye length of each participant, made by a ruler at the end of the experiment. The time window of interest for analysis was 12 s, corresponding to the beginning and ending of the task stimuli. Therefore the mean pupil diameter was calculated by averaging the pupil diameter over the task time window, and the blink rate was estimated as the number of blinks during the task time window divided by 12 s. 4.4 Analysis The dataset is a 2 (perceptual load levels) 5 (cognitive load levels) factorial design, so we employed repeated ANOVA tests in the analysis to examine the main effects and interactions. The critical p value was set as 0.01 and for those within-subject tests that violated the assumption of sphericity, the degrees of freedom were corrected by Greenhouse-Geisser epsilon coefficients, indicated by fractions. Subsequent Bonferroni corrected t-tests were conducted on each task type basis to show which pairs of cognitive load levels were significantly different under each perceptual load condition. Before conducting the ANOVA tests, we subtracted the average pupil diameter during the selected 0.5 s baseline time window to obtain the gain of the pupil diameter elicited by cognitive load (Chen et al., 2013c). After the baseline subtraction, pupil diameter variations due to background changes, different experimental days and different task segments would not affect the task-invoked pupillary response significantly. The mean pupil diameter change over the task duration and over the 30 trials was then used to test hypothesis 3. 16

17 Figure 2. Mean and standard deviation of (a) subjective rating of mental effort, (b) task performance, (c) blink rate, and (d) average pupil diameter change across 22 participants in the arithmetic task (blue) and the search task (red). PL in the legend denotes perceptual load. L in the horizontal axis denotes level. This figure demonstrates that subjective rating and task performance are not sensitive to perceptual load while blink rate and pupil diameter change are, but their patterns are complementary. Note that the task difficulty axis markings are not necessarily the same for the two different tasks; this was done only to allow a compact presentation and for a comparison of the trend from the low to high cognitive load levels. 17

18 5. RESULTS 5.1 Subjective rating of mental effort Figure 2(a) presents the descriptive statistics of subjective rating over the five levels of cognitive load in the two perceptual load tasks. They reveal that increasing the task difficulty results in higher perceived cognitive load in both tasks. A repeated two-way ANOVA test (2 tasks 5 cognitive load levels) suggests that there were significant effects in perceptual load (F(1,21)=8.66, p=0.008), cognitive load (F(2.0,41.6)=190.47, p<0.001) and their interaction (F(2.0,41.6)=32.00, p<0.001). Subsequent Bonferroni corrected t-tests for the comparisons of 1-2, 2-3, 3-4 and 4-5 load level pairs (p value ) revealed that all pairs were significantly different except 2-3 in the search task. 5.2 Performance score Figure 2(b) shows the descriptive statistics of performance scores over the five levels of cognitive load in the two tasks. These match the subjective rating results well. As shown in Figure 2 (b), performance scores declined when the task difficulty was increased in both tasks. A repeated two-way ANOVA test (2 tasks 5 cognitive load levels) found significant effects in cognitive load (F(1,21)=219.44, p<0.001) and their interaction (F(2.2,46.3)=32.95, p<0.001) but not in task type. Subsequent Bonferroni corrected t-tests for the comparisons of 1-2, 2-3, 3-4 and 4-5 load level pairs (p value ) demonstrated that all pairs were significantly different in the search task while only the 3-4, 4-5 pairs had significant differences in the arithmetic task. 5.3 Blink Rate Figure 2(c) reports the descriptive statistics of blink rate over the different types and levels of load. Although there was an increasing trend in the arithmetic task over the five levels of cognitive load, a repeated two-way ANOVA test (2 tasks 5 cognitive load levels) did not show significant effects on 18

19 Figure 3. Average pupillary response across 22 participants to the five levels of task difficulty over 14 s in the low perceptual load arithmetic task (left), and in the high perceptual load search task (right). The task began at the instant of 2 s and the mean of the pupil diameter during s was used as the baseline for subtraction. Then the average pupil diameter was calculated from 2.5 to 14 s for each task type and load level as shown in Figure 2 (d). This figure demonstrates that pupil diameter cannot distinguish cognitive load levels in the high perceptual load task but can in the low perceptual load task. cognitive load and on the interaction of task type cognitive load. However, blink rate was found to be significantly higher in the arithmetic task than in the search task (F(1,21)=35.68, p<0.001). 5.4 Pupil diameter Figure 2(d) shows the descriptive statistics of pupil diameter change over the different types and levels of load. Contrary to the blink pattern that blink rate was lower in the search task, the average pupil diameter was larger in the search task than in the arithmetic task. Meanwhile, the range of pupil diameter change in the arithmetic task was larger than that in the search task. An example detailed pupillary response is shown in Figure 3, where we can see that the highest peak of pupil diameter in the search task is around 0.2mm higher than that in the arithmetic task. A repeated two-way ANOVA test (2 tasks 5 cognitive load levels) showed significant effects on task type (F(1,21)=44.08, p<0.001), cognitive load (F(1.9,39.4)=13.76, p<0.01) and their interaction (F(1.9,26.3)=47.25, p<0.001). Subsequent Bonferroni corrected t-tests for the comparisons of 1-2, 2-3, 3-4 and 4-5 load level pairs (p value ) revealed that only the 3-4 pair was significantly different in the arithmetic task, while no pairs in the search task had significant differences. 19

20 6. DISCUSSION 6.1 Results discussion As observed in Figure 2, the four measures showed distinctive patterns with regard to cognitive load within tasks and to the perceptual load between tasks. The subjective rating of mental effort and performance score were generally found to agree with each other but they reveal little information about the perceptual load since they increase and decrease respectively with cognitive load despite different perceptual load tasks. However, the perceptual load information can be found in the patterns of pupil diameter and blink rate, and their patterns are complementary. Specifically, the significant effects on cognitive load levels in both tasks imply that the relationship between the perceptual load and cognitive load described in the Load Theory of Attention (Lavie, Hirst & de Fockert, 2004) was not reflected in the subjective rating and performance scores, since in both low and high perceptual load conditions, they indicate different levels of cognitive load uniformly. Although the smaller incremental change in the search task might indicate the effect of perceptual load, the significant effect found on the cognitive load under the high perceptual load task leads to the rejection of hypothesis 1. One possible reason for the unexpected variations of subjective rating in the search task is that participants might rate their feeling about the task difficulty or results instead of the actual exerted mental effort. Meanwhile, it was observed with surprise that the performance score can differentiate the cognitive load levels under the situations of high perceptual load, and it may be attributed to the performance criterion. As the chance of correctly identifying each digit is same, 50%, the chance of correctly identifying one number which comprises different digits is uneven under different cognitive load levels. The higher the cognitive load levels, the lower the chance to identify all digits correctly. Since it is difficult to know how many digits in one number were correct, we only can give scores of either 1 or 0 based on whether target strings have been clicked. This is perhaps a limitation of using subjective rating and performance score measures for load measurement. 20

21 With regard to the blink rate, it behaved differently in low and high perceptual load conditions. The blink rate was much higher in the low perceptual load task than in the high, as supported by the positive effect on the task type. It is consistent with studies showing that blink rate decreases when more objects are required to be detected (Van Orden et al., 2001; Chen et al., 2011a) and can be explained as inhibiting blink to maximize the perception (Irwin & Thomas, 2010; Stern, Walrath & Goldstein, 1984). However, blink rate had little variation over the cognitive load levels when perceptual load was high, but an increasing trend when perceptual load was low. The latter increasing trend also confirms previous studies (Veltman & Gaillard, 1998; Chen, Epps & Chen, 2011b; Recarte et al., 2008) which also use an arithmetic task, and can be explained by a relief mechanism (Ponder & Kennedy, 1927), that is, when the mental tension cannot find an internal or external outlet, blink rate increases. If blink rate can index cognitive load levels, then it fits the Load Theory of Attention well, in that cognitive control typically occurs when the perceptual load is low. The negative effect on the cognitive load in the low perceptual load task leads us to tentatively accept hypothesis 2. In searching for an explanation, we speculate that individual differences in mental capacity alter participants information processing. We observed that seven of the twenty-two participants had a decreasing trend of blink rate when dealing with more difficult arithmetic tasks, while others showed an increasing trend. Interestingly, six of the seven participants achieved the top six performance scores in the most difficult arithmetic task. Therefore, the tentative reason could be that the six participants were able to finish the calculating process before the next addend was displayed; whereas other participants were still working on the summation even when the next addend was displayed and its perception was ignored. These two different processes might cause a high achiever in blink rate in the arithmetic task, thus resulting in a negative effect on cognitive load. Further investigation on the difference of groups in blink rate might be needed. In contrast to the effects of perceptual load on blink rate, pupil diameter change was larger during the high perceptual load task than the low, supported by the significant effect on the task type. It seems to indicate that participants actually exerted more mental effort to deal with the high perceptual load task. The lack of 21

22 significant effect on cognitive load in the search task suggests little difference between the five cognitive load levels when perceptual load is high; while the positive effect on cognitive load in the arithmetic task implies a distinguishable magnitude in response to different cognitive load levels when perceptual load is low. This pattern of pupil diameter change seems not identical to the pattern found in the study by Recarte et al. (2008), as they found pupil diameter was still larger in higher load levels in dual tasks. This is very likely due to a much higher perceptual load (overload) in this study, detecting one number from 28 numbers, than theirs, detecting one letter from 16 capital letters. Thus, in this study, the pupil diameter change supports the relationship between the perceptual load and cognitive load in the Load Theory of Attention in that high perceptual load has exhausted participant mental capacity and little available capacity was left to deal with different cognitive load levels. It leads us to accept hypothesis 3. Moreover, the general form of pupillary response over time in the arithmetic task, as shown in the examples of Figure 3, conforms with previous studies (Klingner, Tversky & Hanrahan, 2011; Piquado, Isaacowitz & Wingfield, 2010; Beatty, 1982), in that it gradually increased to a peak with sustained attention. However, in the search task, the pupillary response always quickly reached a peak and saturated, which happened across all load levels, therefore the average pupil diameter change in the search task was much greater than that in the arithmetic task. 6.2 Limitations of the experiment Due to the limitations of this experiment that two different background colors were used, it might be possible to think that small variations between the five load levels of the search task were probably due to a mechanical limitation in pupil dilation in the black background. Our arguments to exclude this mechanical limitation are as follows: (1) Without a task goal, in the baseline procedure (participants merely looked at the countdown numbers), the average pupil diameter across 22 participants from the 2nd second to the end was 4.55 mm in the grey background, and 5.42 mm in the black background. With task goals, the average baseline value (the average values between the two gray vertical lines in Figure 3) across 22 participants for 22

23 subtraction was 4.77 mm in the arithmetic task, and 5.35 mm in the search task. Then the maximum change from the average baseline value was 0.5 mm in the arithmetic task and 0.7 mm in the search task, from Figure 3. If pupil dilation was limited because pupil diameter was already large in the search task with black background, then the range of pupil diameter in the search task should not be larger than that in the arithmetic task. (2) As for the curves from the 1.5th to 2nd seconds in Figure 3, the pupil diameter was presented in an exact order of the five load levels. At this time, the eye was nearly adapted to the black background and participants were trying to memorize the target number, ranging from one to five digits in level one to level five. This demonstrates that pupil diameter can actually distinguish the five load levels in the black background and the search task was more demanding than simply memorizing a five-digit number. Another limitation of this experiment is that we cannot completely separate perceptual load and cognitive load factors as human information processing is a complex process in nature (Maragos et al., 2008). Since stimuli containing a large number of interacting elements held in working memory must firstly be perceived from the visual channel, they inevitably increase perceptual load at the same time as increasing cognitive load. Although the number of digits in the addends in the arithmetic task varied from 1 to 3, we considered these as still low to medium perceptual load. In the search task, by contrast, 28 numbers displayed at the same time almost overloaded participants perception, which was confirmed by informal comments from participants that they did not have enough time to scan all numbers in 3 s before they moved. Even though the number of digits in the target/distractor numbers was changed from 1 to 5 to induce different levels of cognitive load, the load on perception was kept in the range of high to overload. As we intended to observe how low and high (overload) perceptual load affects cognitive load measurement, we think this limitation of a small variation of perceptual load did not change the conclusion. Finally, comparisons of eye metrics would ideally be conducted where variations of perceptual load and cognitive load can be manipulated in the same task. However, we found it is not easy to maintain the level of perceptual load and vary the level of cognitive load in the same task at the same time. Although the level 23

24 of perceptual load was manipulated cross two types of tasks, the blink measure exhibited the same pattern as the number of items changed within a task in previous studies (Van Orden, et al., 2000). 6.3 Implications for HCI In terms of implications for interface design and human computer interaction, knowing the type and level of load that is generating user mental effort will benefit the diagnosis and remedy formulation processes in human-centered design. Imagine an interface where there are more distracting features than the useful ones, like the search task in this study, users will experience high perceptual load when looking for useful information. Altering the volume of useful information will not help reduce the user s mental effort. If this type of load is diagnosed, distractive interface features could be removed to minimize the extraneous complexity (Oviatt, 2006). In another case where the perceptual load is low, like the arithmetic task, users may depend heavily on working memory to engage in problem solving or planning. If interfaces can provide corresponding support to facilitate their cognition, for example, minimizing interruption, providing alternative input modalities (Oviatt, 2006), the intensity on working memory may be offloaded to some extent during complex tasks. One implication from this study is that subjective rating and performance score, as two main measures used, on their own are insufficient to tell the type of load in which the mental effort occurred, if the type of load is also of interest. This may be important for user interface evaluation methods that use these metrics. A suggested method is to obtain blink rate and pupillary response since blink rate can differentiate low and high perceptual load while pupil diameter can indicate both high perceptual load and high cognitive load during tasks. We illustrate two examples based on previous studies to demonstrate the practical applicability of the results. The first one is from the Goldberg & Kotval (1999) study in the context of selecting good and poor interfaces for file editing. The functionality of the interface is to provide 11 tools, whose buttons displayed on the left side of the work area. In the poor interface, all tool buttons were randomly displayed 24

25 in one area. Therefore, the perceptual load is generally high because users have to perceive the icons on each button to find the desired one. In the good interface, tool buttons are functionally grouped into three categories: editing, drawing and text manipulation, and graphically displayed in three areas. Therefore, users only need to search the desired tool in one of the areas and the perceptual load is relatively low. In the Goldberg & Kotval (1999) study, eye movement locations and scanpath were suggested to assess interface quality. However, it may not be suitable for interfaces where specific features are unknown, and it may also suffer from interpretation difficulty because it is hard to identify attention and intention from eye movement when tasks are more complicated and require longer time to complete (Jacob & Karn, 2003). Using pupil diameter and blink rate during a formal eye activity-based evaluation should be able to overcome the above limitations or add another dimension to assess interfaces regardless of what or how much the users want to edit or draw in this interface. This is because they are covert responses to the overall interface features when using them during tasks. In this example, whatever the editing content in working memory is (cognitive load level), users will experience higher perceptual load in the poor interface. The second example is from the study by Oviatt (2006) in which three different user interfaces paperbased, pen-based, and tablet-based were compared for math education. In this study, task performance, reaction time and questionnaires were used to assess which kinds of interfaces benefit learning or problem solving. In this example, utilizing pupil diameter and blink rate might be a more direct method for interface evaluation because, according to our study results, if users are experiencing high perceptual load during tasks, it obviously does not help facilitate cognitive processing. Application designers could artificially change task difficulty levels (cognitive load) executed on each interface, and then detect which interface is better in terms of perceptual load by observing the pupil diameter and blink rate. If pupil diameter and blink rate can identify this type of source of errors in interfaces, designers attention should be paid to how modality or instruction design helps reduce load rather than how to display math problem instructions. 25