Modeling and Implementing an Adaptive Human-Computer Interface Using Passive Biosensors

Transcription

1 ing and Implementing an Adaptive Human-Computer Interface Using Passive Biosensors Curtis S. Ikehara, David N. Chin and Martha E. Crosby University of Hawaii at Manoa Department of Information and Computer Sciences Abstract ing of the human-computer interaction as a partnership between two systems provides a flexible method of modeling both the quantitative and qualitative requirements of a human-computer interface. A discussion of the components of a system, system structures and system issues are reviewed along with a description of the research model used at the Adaptive Multimodal Interactive Laboratory. 1. Introduction For a partnership to achieve both short and long term project goals, good communications, sufficient combined abilities and the availability of adequate resources are necessary. Each person has a variety of goals, abilities, plans and ways of responding to change. Good communications is necessary to: establish common goals, identify abilities, identify resources, generate plans, identify the quality of the outcomes, and modify plans to improve the outcome or modify plans when goals, abilities or resources change. Each person learns from the other and each builds a model of the other person. Using the constructed model, each person acts to obtain an optimal outcome through cooperation and compromise. When a person works on a computer, a humancomputer partnership is established. Both user and computer are separate interacting systems. The user is tasked with building a model of how the computer functions and the computer by design has a preset model of how the user functions. The approach at the University of Hawaii Adaptive Multimodal Interaction Laboratory (AMI) is to use information obtained from passive physiological sensors (i.e., biosensors) monitoring the user to modify the actions produced by the computer's user model to optimize the outcome. This paper will discuss the modeling of the human-computer partnership as the interaction of two systems and will include a discussion of the components of a system, system structures, system issues and how these features are being integrated into the research model at the AMI Laboratory. The benefit of this modeling approach is that it provides a flexible method of modeling both the quantitative and qualitative requirements of a humancomputer interface System Components and Construction Figure 1 shows the common global diagram of the human-computer interaction with user and computer interacting. The human-computer interaction shown in Computer System User System Figure 1. Human-computer interaction. Figure 1 is a closed loop system that will be described in greater detail later in this section. There are two basic system types: open and closed loop systems. Figure 2 shows a generic open loop system. The goal is the desired outcome. The model contains the a priori knowledge of what will be the result of a set of actions and how those actions are chosen. Actions taken are those that are assumed to bring the outcome closer to matching the goal. The system is what is being acted upon. The system could be either a user or a computer as shown in Figure 3 and 4. The outcome is the result of the actions on the system, but the goal and the outcome may not necessarily be the same /04 $17.00 (C) 2004 IEEE 1

2 Actions System Outcome Figure 2. Generic Open Loop System. User Actions User Outcome Figure 3. Open loop computer system, where the computer contains a model of the user and performs actions on the user. Computer Actions Computer Outcome Figure 4. Open loop user system, where the user contains a model of the computer and performs actions on the computer. Actions System Outcome Figure 5. Closed loop system. With the open loop computer system in Figure 3, the computer has a goal and a model of the user imbued by the computer system designers. The goal is usually to receive an instruction and perform a predefined sequence of actions for the user resulting in the desired outcome (i.e., the goal criteria). can significantly influence the outcome of a user, but in this diagram the computer has no mechanism to assess the influence of noise on the outcome. With the open loop user system in Figure 4, a user has a goal and a model of how the computer functions. The user combines the goal and model to generate actions that are applied to the system and result in an outcome (i.e., output). is not a significant factor regarding the outcome from actions taken on a computer system. When the user system has a good model of the computer system the actions taken by the user will be Measurement Error appropriate for the computer system to produce the desired outcome. In the open loop system, it is assumed that the actions applied to the system will produce the desired outcome, but this is not always the case. When the outcome does not meet the goal criteria another set of actions are required. Detecting the difference between a goal state and an outcome requires the introduction of a sensor. Figure 5 shows a closed loop system with the sensor sending information about the system to the model. The advantage of the sensor is that an error can be detected and the model can take actions to reduce the error. The disadvantages are that the model requires more available resources to handle this extra input. A sensor with limited accuracy and response time could introduce significant measurement errors which would have a detrimental effect on the outcome and increase demands on the model s available resources /04 $17.00 (C) 2004 IEEE 2

3 Actions System Outcome Figure 6. Closed loop system with second sensor Figure 7. Multiple goal system. Actions System Outcome Action Sub- System Outcome Subsystem N G M A S-S Out Figure 8. A subsystem within a system Sen Ext Actions Ext A sensor may only provide measures relating to one criterion of a goal. Figure 6 shows a model with a second sensor to monitor a different goal criterion. Multiple sensors may be necessary to accurately determine the difference between the goal and outcome. A system model can have multiple goals that result in a single outcome as shown in Figure 7. Besides the increase in model resources required to accommodate the increased number of inputs to the model, it is usually necessary with multiple goals to create a heuristic to prevent goals from conflicting. For example, optimizing speed and accuracy simultaneously are two conflicting goals that commonly require prioritizing. Figure 8 shows a system (i.e., subsystem) within a system. The subsystem model combines the external actions with the subsystem goal to produce actions on the subsystem of the subsystem. Denoted by S-S in Figure 8. The S-S response to subsystem actions are externally sensed by the system. Figure 8 could be redrawn to Figure 9 if S-S is drawn to be the system. This would make Figure 9 similar to the interaction between computer and user of Figure System Issues In the introduction, partners construct models of each other as part of a process to optimize the outcome of the partnership. Problems occur when a /04 $17.00 (C) 2004 IEEE 3

4 Computer System Computer User Action desired task performance improvements would be for the user to be more accurate, faster and make strategic decisions (i.e., situationally appropriate). "Augmented cognition" is the name of the research area that encompasses the methodologies used to achieve these goals. User System User Computer Figure 9. A subsystem within a system Action preconceived model is inappropriate, a model is based on incorrect information, or a model is based on insufficient information. In the first and second cases, the system model will not produce the desired outcome or be very erratic. In the third case, the lack of sufficient information about the range of conditions may cause a system model to work only under specific conditions and not be robust enough for general use. Other component factors of the system can have a significant impact. Both actions and sensors can be noisy, low in accuracy, and slow in response. The effect on the system and model respectively could detract from achieving the desired outcome and demand more available model resources. The model may require more resources to handle additional inputs from sensors and goals. Should there be insufficient resources, degradation away from the desired outcome would occur. The designer of a system must consider the goals and assemble a system that functions over the range of changing goals, noise and measurement error. It is important to note that it is common for a user to construct and use models that are situationally functional and are superceded by other models more suited to the situation. 2. Experimental Methodology The primary goal of the research at the AMI laboratory is to create a methodology to improve learning and task performance by optimizing the human-computer interface based on the user's cognitive state which is obtain from passive physiological measures (i.e., biosensors). The desired learning improvements would be to increase the rate of learning, comprehension and retention while maintaining or increasing user satisfaction. The 2.1. Adaptive Information Filter Figure 10 shows a detailed diagram of the computer and user system used for augmented cognition research. The Moving Target Fractions (MTF) task and adaptive information filter are subsystems of the computer model of the user. What makes this model unique is that there are biosensors (i.e., passive physiological sensors) that monitor the user and provides input to the adaptive information filter. Research is underway to determine which sensor or combination of sensor data will be necessary to assess the user's cognitive states listed in Table 1. Once the cognitive state of the user is assessed, presentation changes are made by an adaptive information filter to improve task performance Biosensors The first column of Table 1 lists all the physiological sensors currently being used (i.e., biosensors). Secondary measures (column 2) can be extracted from the primary physiological measures and are needed to derive some of the potential cognitive and affective states. Column 3 list several potential cognitive measures. Eye fixation duration and saccade length, derived from eye tracking, has shown that background distracters increase processing time, increasing the number of distracters increases search time and fixation duration and saccade length are related to background complexity [4]. The mouse pressure measurements are derived from a computer mouse equipped with sensors that detect the forces applied to the mouse when clicking or squeezing the mouse. Preliminary results indicate that forces applied to the mouse are correlated to task difficulty. Skin conductivity, peripheral temperature, and heart rate have been shown to be related to mental states such as arousal, relaxation, and stress [1][4]. The design of the MTF task allows computer mouse position and mouse speed to be indicators of both mental and cognitive states. Increased motion can indicate stress or arousal and as the task grows more difficult the mouse position shifts more towards the right edge of the screen /04 $17.00 (C) 2004 IEEE 4

5 Computer System User System s (see below) s (see below) Actions Modified Text Modified Graphics Image Eyes Computational Visual/Spatial of the Computer (the user s belief of how the MTF computer program functions) of the User MTF Actions to Complete Task Adaptive Filter How the presentation should change based on biosensor data. How the presentation should change based on task performance Keyboard Mouse Cognitive Activity (mental workload & emotional impact) Biosensor Signals Biosensor Task Associated Physiological Activity System Designer s Learning s 1. Increase the rate of learning 2. Increase comprehension 3. Improve retention 4. Maintaining/increasing user satisfaction Task s 1. More accurate 2. Faster 3. Making strategic decisions Figure 10. Detailed computer and user system interface. Primary User : Highest possible score Subgoals: Prioritize and select 1. Evaluate all fractions as they appear to determine if the fraction can be evaluated quickly or with difficulty 2. Evaluate each fraction's value, within the user's confidence level, to determine its relationship to a critical value. 3. Consider how the score is computed when selecting targets 4. Not let a fraction greater than the critical value touch the right edge 2.3. Testbed Software Description Moving Targets Fraction Task The testbed software, MTF (Moving Targets Fractions), presents a controlled cognitive load task to the user and adapts the presentation by adjusting the degree of information filtering based on what the biosensors indicate is the instantaneous cognitive load of the user (i.e., adaptive information filtering). The MTF task presents on a computer screen a fixed number of oval targets containing fractions. These fractions float across the screen from left to right (see Figure 12). The motion and scoring gives the MTF task a video game flavor that helps maintain the user's interest. MTF Visual Input/Feedback (Eyes) Subgoal #1 Subgoal #2 Subgoal #3 Actions Subgoal #4 Figure. 11. User system when interacting with the MTF task /04 $17.00 (C) 2004 IEEE 5

6 Table 1. Measures from the Biosensor and Potential Cognitive States Physiological Measures Eye Position Tracking Secondary Measures Gaze Position, Fixation Number, Fixation Duration, Repeat Fixations, Search Patterns Potential Cognitive Measures Difficulty, Attention, Stress, Relaxation Problem Solving, Successful Learner, Higher Level of Reading Skill [1][2] Pupil Size Blink Rate, Blink Duration Fatigue, Difficulty, Strong Emotion, Interest, Novelty, Mental Activity - Effort, Familiar Recall, Imagery, Abstract vs. Concrete Words, Language, Processing, Affective Words, Shocking Photos, Positive / Negative Attitudes, Information Processing Speed [1] Skin Conductivity Tonic and Phasic Changes Arousal [1] Peripheral Temperature (Finger, Wrist and Ambient) Relative Blood Flow Mouse Pressure s (Left/ Right Buttons and Case) Heart Rate and Beat to Beat Heart Flow Change Negative Affect (Decrease) [4], Relaxation (Increase) [1] Stress, Emotion Intensity [1] Stress [3], Certainty of Response Mouse Position Speed of mouse motion Arousal, Stress, Problem Difficulty Maximizing the Score and Subgoals Figure. 12. Screen capture of the Moving Targets Fraction (MTF) task. The primary goal of the user is to maximize the score by selecting the correct fractions before they reach the right edge of the screen. Cognitive load of the MTF task is controlled by adjusting fraction values, speed of the fractions across the screen and the number of fractions presented. Cognitive load is also affected by changes in the task requirements needed to achieve the primary goal. Adaptive information filtering provides incomplete but helpful information to the user to reduce cognitive load and the degree of filtering is modified based on the user s cognitive state. For the user to obtain the highest score, the user must select all fractions greater than the critical value of 1/3 before they touch the right edge of the screen. The goal of the user is to maximize the score, which is prominently displayed at the bottom of the screen, by achieving four subgoals before taking action. The first subgoal is to evaluate all fractions as they appear to determine if the fraction can be evaluated quickly or with difficulty. The second subgoal is to evaluate each fraction s value, within the user s confidence level, to determine its relationship to a critical value. Fractions that are greater than the critical value will increase the score when selected. The difficulty of the comparison (i.e., cognitive load) is controlled by the selection of fractions. For example, 1/2 is obviously greater than 1/3, but comparing 6/17 versus 1/3 requires much more cognitive effort. The user registers decisions by clicking with the mouse on those fraction targets greater than the critical value. For even greater difficulty, the critical value can be changed from simple fractions like 1/3 to complex fractions such as 5/13. The third subgoal is to consider how the score is computed when selecting targets. The score is /04 $17.00 (C) 2004 IEEE 6

7 computed as 100 times the fractions that the user selects correctly above 1/3 (e.g., 3/4 * 100 = +75) and deducts 100 points for each incorrect selection. The negative scores for incorrect targets means the user cannot simply select everything on the screen to maximize the score. The fourth subgoal is to not let a fraction greater than the critical value touch the right edge. A deduction of 200 times the fraction value will occur if a fraction greater than the critical value touches the right side of the screen while a score of 200 points are added when the fraction is below the critical value. This motivates the subject to evaluate all fractions presented and not just the easily computed ones. The subgoals can take on different priorities depending on task variables such as the difficulty of evaluating the fraction, the value of the fraction, how close the fraction is to the right side of the screen and the number of fractions presented. The priorities of the subgoals can also be affected by user factors such as arousal, stress and motivation Cognitive Load As mentioned in section 1.1, having several subgoals increases demands on the model s available resources by increasing the input to the model and requiring a heuristic to deal with prioritization. In the MTF task, there are four subgoal inputs in addition to a complex heuristic where the prioritization of the subgoals change according to the situation. Computational load is generated by the nature of the fraction value, number of fractions presented, subgoals and heuristic required. Also mentioned in Section 1.1 is that increasing the number of sensors or increasing the measurement error of a sensor would increase the demand on the model s available resources. For the MTF task, visual/spatial load is affected by the number of standard appearance fraction ovals presented which increases the sensor requirement and movement speed of the fractions which increases the potential for error. There are two types of cognitive ability of interest when performing the MTF task. The first cognitive ability of interest is numeric computation since the task requires the user to estimate and compute fraction values in relation to a critical value. The second cognitive ability of interest is visual/spatial load that occurs as the user is tracking the multiple fractions as they float across the screen. The biosensors can provide a measure of the user s cognitive load inferred from the cognitive measures and mental states detected (see Table 1). The potential for both computational and visual/spatial load assessments in real-time are possible with the biosensors. Computational load could be extracted from blink rate, blink duration, stress levels, and arousal state. Visual/spatial load could be extracted from gaze locations, search patterns and eye fixations. A user's maximum cognitive ability is indirectly determined by increasing the cognitive load (i.e., task difficulty) until there is a significant change in the error rate. Initially, to determine a user s maximum cognitive ability and calibrate the biosensors, a user would complete a set of MTF tasks with various levels of difficulty to identify at what level the error rate significantly increases. It is desirable to have the cognitive load of a user within a range below the user s maximum cognitive ability, but above a minimum level. The upper limit of the range is selected to minimize errors or to maximize the score for the MTF task. Considerations such as the length of the task can affect the location of the optimal range. For example, a multiple event task taking several minutes might have an optimal range considerably below the maximum cognitive ability in order to avoid burnout whereas a single event task taking less than a minute might best be handled at or very close to the maximum cognitive ability. A cognitive load lower limit is useful for maintaining motivation and minimizing a decline in cognitive ability due to boredom. Each type of cognitive load may have a different optimal range because of inherent differences in the long-term sustainability of the cognitive ability and its affect on the user's overall mental state. In Figure 13, Cognitive Load #1 (Computation) has a graded vertical scale. The top of that scale represents the maximum cognitive ability. When the user is performing the MTF task, each subgoal (i.e., S#1, S#2, S#3 & S#4) increases the total computational cognitive load (i.e., Load #1 ). The solid arrow next to Load #1 shows the current computational cognitive load. The dash arrow indicates a desired cognitive load value. The desired computational cognitive load range is denoted by the two headed arrow. The figure shows the computational cognitive load is currently above the desired range. In the same figure, Cognitive Load #2 (Visual/Spatial) shows the visual/spatial cognitive load to be less than the desired range. How the adaptive information filter shifts cognitive load from computational to visual and maintains only the computational cognitive load within the desired range is described below Information Filtering Information filtering can be presented to the user by a combination of three methods: emphasis, deemphasis and deletion. In the context of the MTF task, more important information (high value targets) can be /04 $17.00 (C) 2004 IEEE 7

8 emphasized (highlighted) by making it bolder, larger, in brighter colors, animated, and/or displayed with sound effects. This shifts information that is not being processed by an overloaded cognitive ability (e.g., computation) to a less loaded cognitive ability (e.g., visual or audio processing). Second, information presented to the user can deemphasize low value targets (e.g., fading, shrinking, etc.). Third, low value targets can be omitted. A completely autonomous program that could perform the MTF task perfectly would contain all of the specifications of the system, monitor the task and output the appropriate actions. Changing goal priorities, incomplete system specifications, sensor measurement error and outcome inaccuracy are why a computer will not be able to complete real world tasks adequately and tools like the information filter will be useful to facilitate task completion. This lack of intelligence of the adaptive information filtering program is simulated by giving the adaptation algorithm only part of the Max S#4 S#3 S#2 S#1 information necessary, the information needed to solve subgoals 1 and 3, but not subgoal 2 and 4. Thus the adaptation algorithm does not know which targets are really correct; it only has the knowledge that certain targets are better than others based on the relative values of the fractions. As mentioned in section 1.1, fewer subgoals place less demand on available resources. Information filtering using de-emphasis is the preferred method since it will allow an incremental change in the task difficulty. For example, if there were 10 fraction targets on the screen to compute, the user must evaluate 10 targets for computational ease or difficulty (i.e., subgoal #1). Deemphasizing by fading would change the task to evaluate nine unchanged targets first, then the faded target. The problem with emphasizing is that it can be distracting or can significantly change the task. Emphasizing nine targets by highlighting would be visually distracting and highlighting a single target would change the task to evaluate the highlighted fraction first, then the remaining nine. Removing targets is not as desirable since it may reduce the user s score when the user tries harder than the system anticipates. For the MTF task, the user is operating below the maximum cognitive ability and by increasing cognitive effort beyond the MTF Display Output Adaptive Information Filtering Module (Deemphasize a number of low value targets based on optimizing Load #1 and Load #2) Cognitive Load #1 (Computation) Load Before Load After Ideal Range Max S#3 S#1 Visual/ Spatial Biosensor Data MTF Display Input Cognitive Load #2 (Visual/Spatial) Figure. 13. The adaptive information filter changes the MTF display depending on the computational cognitive load derived from the biosensors. Load Before Load After Ideal Range optimum range could obtain a maximum score without the information filtering. Removing targets may be best for those users that refuse to pace themselves and always work at their cognitive limit. A non-adaptive information filtering module of the MTF program would filter out a constant number of the lowest scoring fractions, allowing the user to concentrate on the highest scoring fraction targets. An adaptive information filtering program that changes the number of filtered targets based on the user s computational cognitive load, as measured using the biosensors, should perform better than a constant filtering rate since a constant filtering rate will filter too many targets when the fractions are easy and too few when the fractions are difficult for the user. Only the computational cognitive load is used to adjust the adaptive information filter since using both computational and visual/spatial cognitive load measures may produce conflicting signals to the adaptive information filter. The adaptive information filtering program would improve performance by maintaining cognitive load in the optimum range which will minimize user errors (i.e., short term goal) and preserve the user s maximum cognitive ability (i.e., long-term goal). The biosensors data interpreted by the information filter can be used to verify that the information filter /04 $17.00 (C) 2004 IEEE 8

9 shifts computational cognitive load to visual/spatial cognitive load as shown in Figure 13 (i.e., Load Before & Load After ). When the adaptive information filter increases the number of targets filtered to maintain the computational cognitive load within range, the visual/spatial cognitive load should increase. Although this is clearly a contrived task, it is similar to many real world decision tasks that require selection of targets under a time pressure. For example an air traffic controller may benefit from filtering of non-critical planes when numerous planes significantly increase the error rate. Software could fade the intensity of non-critical planes while maintaining the intensity on those planes that are on potentially dangerous courses (e.g., will intersect with another plane, will run out of fuel soon, etc.). Of course, the user must still make the final decisions since software is not intelligent enough to do the job adequately without the user. 3. Conclusion The augmented cognition research at the AMI laboratory is targeted at collecting biosensor information from the user so that the adaptive information filtering program can in real-time optimize the presentation of information and achieve learning and performance goals. The MTF task requires the user to elicit several important abilities. These abilities include: hand-eye coordination, visual search, mathematical computation, fraction estimation, strategy selection, learning and motivation. All these requirements of the task affect the user's cognitive ability and constrain cognitive load to different degrees. Manipulation of the presentation to the user is designed to control the user s cognitive load and optimize the user s cognitive ability to achieve the short term goal of maximum performance and the long term goal of maintaining a high level of cognitive ability. The qualities of a productive human-computer interaction are the same qualities that a productive partnership should have. Those qualities involve effective communications between the partners to combine the activities and resources of the partnership to efficiently solve immediate problems while strategically addressing goals and growing new capabilities. The basic components of a model in the first section provided a methodology for constructing the detailed human-computer interaction model shown in Figure 10. In the future, application of the methodology can lead to a model of a productive human-computer interaction with greater detail that can guide experimentation and provide greater explanatory power about the human-computer interaction. 4. References [1] J. L Andreassi, Psychophysiology: Human Behavior and Physiological Response, Third Edition. Hillsdale, NJ: Lawrence Erlbaum, [2] E. Sheldon, Virtual Agent Interactions, Doctoral Dissertation, Orlando: University of Central Florida, [3] C. Lange Küüttner, Perceptual and Motor Skills, 86 (3 Pt 2), 1998, pp [4] M. E. Crosby, M. K. Iding and D. N. Chin, Visual search and background complexity: does the forest hide the trees?, In: Bauer, M., Gmytrasiewicz, P. J., and Vassileva, J. (eds.): User ing 2001Springer-Verlag, Berlin Heidelberg New York, Acknowledgments This research was supported in part by the Office of Naval Research grant no. N and DARPA grant no. NBCH /04 $17.00 (C) 2004 IEEE 9