The Role of Cognitive Strategy in Human- Computer Interaction. The Premise of This Talk. Efficiency inspires delight.

Transcription

1 The Role of Cognitive Strategy in Human- Computer Interaction nthony Hornof ssociate Professor Computer and Information Science University of Oregon Collaborators: David Kieras, David Meyer, Tim Halverson, Yunfeng Zhang Sponsorship: NSF, ONR January, 06 The Premise of This Talk The field of human-computer interaction needs better predictive engineering techniques. These engineering models should be based on a specification of the user, the task, and the device. The specification of the user should be based on how humans process information. People use cognitive strategies plans that coordinate the perceptual, motor and memory processes needed to do tasks. Efficiency inspires delight. esthetics and usability (as measured by speed and accuracy) both influence a user s feelings about an interface. Usability influences it more. (Thüring & Mahlke, 007; Tuch et al., 0) User interfaces that reduce the processing required to accomplish tasks are perceived as more aesthetic and appealing. (McDougall & Reppa, 0) Core Research Question: What are the cognitive strategies that people use (and do not use) when interacting with computers? Cognitive psychologists have asserted such strategies for decades: Shiffrin & tkinson (969) - control processes to modulate information flows between processors. Baddeley (97) - executive processes. Lachman et al. (979) - the programs in the information processor. Rosenbaum (99) - motor programs. But few researchers have studied them. They are elusive. Data and techniques that I use to expose the cognitive strategies: Cognitive modeling. Eye.

2 The The EPIC Cognitive rchitecture (Kieras and Meyer, 997) rchitecture (Kieras and Meyer, 997) Task Environment Simulated Interaction Devices uditory Input Visual Input Long-Term Memory Production Memory uditory Processor Visual Processor Ocular Motor Processor Vocal Motor Processor Manual Motor Processor Cognitive Processor Production Rule Interpreter Working Memory Tactile Processor The EPIC Cognitive rchitecture EPIC: Executive Process-Interactive Control Kieras and Meyer (997) Inspired in part by the Model Human Processor (Card, Moran, and Newell, 98) Captures human perceptual, cognitive, and motor processing into a simulation framework Constrains the models that can be built Inputs into the architecture: Task environment Visual-perceptual features Cognitive strategies Outputs from the running model: Execution times Trace of the processing Eye movements 6 Cognitive strategies are simulated with production rules. n EPIC production rule to move the eyes. (from Halverson & Hornof, 0) The visual retinal zones in EPIC 97 center of gaze parafovea bouquet fovea periphery not in view Visual features (such as text, shape, color) are available as a function of the angular distance between an object and the point of gaze. 7 8

3 EPIC is well-suited to explore cognitive strategies EPIC makes an architectural commitment to only enforce serial processing for motor activities. ny number of production rules can fire in each 0 ms cycle. Strategies can be written to permit one rule to fire at a time (as in our initial model) and also to explore the potential for overlapped strategies (as in our second and third models). Four Projects Exploring Cognitive Strategies for. Visual search of pull-down menus.. Visual search of hierarchical displays.. Multimodal multitasking.. Individualized strategies for multitasking Cognitive strategies for visual search of pull-down menus. Menu search and selection remains a dominant HCI interaction technique and yet the field still lacks a comprehensive predictive model of how people do it. It is not a solved problem. Visual Search of Pull-Down Menus Data from Nilsen (99) Models from Hornof & Kieras (97 & 99) Menus:, 6, or 9 numerical digits. Randomly or Numerically ordered menus. Blocked by size and ordering. Procedure: Study target digit. Click on "GO". Menu appears. Point to and click on target. GO GO 6 Randomly ordered 6-item menu

4 Human data Explain the data by exploring strategies Main effects: Numericallyordered menus are faster. Different slopes. In Randomly ordered: Position effect. Menu length effect. Selection Time (msec) Randomly ordered menus Numerically ordered menus Fitts law prediction Serial Position The goal is to explore strategies while keeping the architecture constant. Exploring strategies presents two problems:. Degrees of freedom.. Falsification. These are addressed in part by:.proposing strategies that (a) are parsimonious (at least debatably), and (b) have been proposed and debated in the literature..ttempting to falsify some of the competing theories within the context of the model exploration. Serial Processing Models Disconfirm strategies for searching randomlyordered menus. Serial processing strategy proposed by Kent Norman (99) Scanning Process Is list exhausted? Find next Item Encoding Process Matching Process Mismatch Match Select Failure Random Search (Lee & MacGregor, 98) Selection Time (msec) G-00T Serial Position Systematic p-to-bottom Search Search (Card, 98) Selection Time (msec) G-T Predicted Predicted Observed Observed Serial Position Parallel processing strategies These steps take place in parallel. Look at the precue. Click the mouse on the GO box to show the menu and move eyes to where one of the top items will appear. Determine the item one foveal diameter below gaze. Move eyes to that item. Quit searching when target item appears in working memory. Move mouse and gaze to item. Click mouse. 6 0

5 mixture model of random and systematic, and different numbers of items perceived in parallel, explained the data Dual Strategy / Varying Distance Hybrid Model Selection Time (msec) Hybrid /8 When you generate strategies that use the architecture as effectively as possible, you can predict some interesting behaviors Maximally-efficient foveal sweeps Parallelism built into the architecture permits multiple items to be considered in parallel Serial Position 7 8 Cognitive strategies are identified using reaction time data and the architecture Eye movement data subsequently validated many aspects of the strategies Selection Time (msec) Serial Position CHI 97 models for randomly ordered menus, E % CHI 99 models for numerically ordered menus, E % Data Models 9 People process multiple items in parallel. Scan paths are sometimes top-to-bottom, sometimes random, and sometimes a combination of the two. (altonen et al., 998; Byrne et al., 999) Eye tracking and cognitive modeling have a special synergy: The models provide specific theories that the data can confirm or disconfirm. 0

6 Four Projects Exploring Cognitive Strategies for. Visual search of pull-down menus.. Visual search of hierarchical displays.. Multimodal multitasking.. Individualized strategies for multitasking.. Cognitive strategies for the visual search of hierarchical displays This project demonstrates how people clearly use different strategies for different tasks. The Experimental Task Human Data C E B of visual angle D F Experimental Design X design. Layouts were labeled or unlabeled. Layouts had,, or 6 groups. Blocked by layout type. Procedure: Study precue, click on precue, find target, click on target. 6 participants, motivated to search quickly Trends: (a) Smaller layouts are faster. (b) Labeled layouts are faster. (c) Unlabeled layouts have a larger number-of-groups effect.

7 mixed group-label search. Task analysis and the observed search time Unlabeled layouts: Systematic search strategy Figure. EPIC s visual space during a noisy systematic search of an unlabeled six-group la The target is in Position, near the top of the third column. If the fovea had missed the targ this sweep, it would have picked it up on a subsequent sweep. Unlabeled layouts: Random search strategy B C D E F Target Group gain, neither a purely systematic nor a purely random search explain the data. B C D E F Target Group 6 Figure. Unlabeled layout search times predicted by the noisy systematic search strategy. EPIC s visual space during a noisy systematic search of an unlabeled thesix-group saccade distributions layout. that best explain the data. Initial fixations were distributed among get is in Position, near the top of the third column. If the fovea had missed the target on Unlabeled layouts: noisy-systematic search strategy tions,,,, & and The subsequent noisy-systematic fixations among search items strategy,, 6, & explains 7 below. The model pre ep, it would have picked it up on a subsequent sweep. the data with an E of 6.0%. Effectively combines random and systematic within a single strategy. ttempt to make a maximally-efficient foveal sweep, but it sometimes overestimates how far the eyes can move still foveate everything, sometimes missing. the reaction time data for unlabeled layouts 7 B C D E F Target Group 8

8 Labeled layouts ssume that people search the group headings until they find the target group, and then search within the target group. Random search of the group labels Mostly systematic search of group labels Better predictive tools are needed, and the tools need to use cognitive strategies. The Display nalysis Program (DP, Tullis 988) did not incorporate cognitive strategies and could not explain the data. EPIC with the strategies shown. Display nalysis Program (988) B C D E F Target Group B C D E F Target Group Search Time (ms) R T D Observed Predicted Unlabeled Labeled Target-Only 9 Number of Items in Layout 0 The models were built based on reaction time data and subsequently confirmed with eye tracking data. The models were built based on reaction time data and subsequently confirmed with eye tracking data. Unlabeled Layout Labeled Layout

9 Count The cognitive strategies predicted the characteristics of the eye movements at the start of a trial nticipatory Fixations nticipatory Fixations P Layout Position Second Fixations Respond to layout onset Time after precue appeared (ms) Cognitive strategies for visual search should: Optimize architectural constraints. Make an initial anticipatory fixation. Respond to the onset of the layout. Examine more than one item with each fixation. Stop when the target appears in visual WM. Four Projects Exploring Cognitive Strategies for. Visual search of pull-down menus.. Visual search of hierarchical displays.. Multimodal multitasking.. Individualized strategies for multitasking.. Multimodal multitasking Scientific goal: Develop a rigorous, accurate, reusable theory of the cognitive strategies and information processing involved in multitasking. 6

10 Multimodal Dual Task Practical Goal: Predict Human Performance Present emergency workers with multimodal devices that support secondary tasks while personnel are engaged in other life-critical primary tasks. Tactical Classification Task version of the task developed by Ballas et al. (99) at the Naval Research Laboratory (NRL). Task CDispatch.com NYTimes NJ.com H N chproducts.com 8 Peripherally visible, Sound off Two experimental conditions:. Peripheral visible or not With peripheral visible, the contents of both displays are visible all the time. With peripheral is not visible, only the blips on the display that you are currently looking at. Simulates two displays far apart.. Sound on or off When sound is on, an alarm indicates the color change that signals that a blip is ready to classify. 9 0

11 Peripheral not visible, Sound on The eye movement data reveal many aspects of multitasking The time preceding eye movements across the lifetime of a blip after it becomes ready to classify. to Keypress is the how long the eyes are back on the tracking display before a blip is keyed in. Time Preceeding Preceding Movement (seconds) Eyes Radar Eyes Target Time On Blip Keypress Eyes Radar Eyes Target Time On Blip Keypress Movements to Classify a Blip Peripheral Not Visible Peripheral Visible Two strategies will be presented Independent Subtask Strategy Enforces mutual exclusion between the two subtasks. Strict serial processing of each subtask. Do dual task. Independent Subtask Strategy If a blip is ready to classify, do tactical.. Independent Eye and Hand Strategy Select blip to classify Look at blip Get blip features Key-in response Determine if a blip is ready to classify If no blips are ready to classify, do tracking. Check for auditory alarm or visible change in blip. If no peripheral visibility or sound, and time has passed, move eyes to tactical. Move eyes to tracking cursor If tracking cursor is not green, move joystick.

12 Observed Model Ey Ey es T es o R ki Ti ad ng m T a e O arg r e K Ey e n B t y l Ey es T pre ip Tr es o ss ac R ki Ti ad ng m T a e O arg r Ke n B et yp lip re ss Tr ac Movements to Classify a Blip ss ip et re yp Ke n O e Ti Tr ac ki ng Ey Bl ar rg Ta R es es Ke Ey m ad ss ip yp re Bl rg n Ta O e Tr ac ki ng Ey Ti es es m ad ar et Red/Green Yellow Predicts reaction times (E %) but not eye movements (E 9%) or tracking error (E %) from Ey Predicts reaction times (E %) R Red/Green Yellow Preceding Movement (seconds) Time Preceeding Classification Time (seconds) Classification Time (seconds) Model Peripheral Visible Red/Green Yellow Observed Peripheral Visible Red/Green YellowRed/Green Yellow Peripheral Visible Peripheral Visible Model from Peripheral Not Visible from Peripheral Not Visible Peripheral Visible Observed Peripheral Visible Classification Time (seconds) Red/Green Yellow Movements to Classify a Blip 6 Independent Eye and Hand Strategy Key: Task switching Perceptual information ss re Ke m e O n yp Bl ip et rg ar g in ck s Ti Ta ad R Ey e es Ey Ke n yp re Bl ss ip et rg O Ti g in ck Predicts reaction times (E 7%) and eye movements (E 0%) and tracking error (E %) e R keypress m Red/Green Yellow Tr a Subtask delegation Red/Green Yellow Resume when the blip is classified. s move joystick Classification Ey e Tr a Interrupt when a blip is ready to be keyed in. Ta blip features Manual Motor es tracking cursor color Look at blip Ey Look at tracking target ar Resume after the active blip's features are encoded. ad Classification Peripheral Visible 7 State transition diagram summarizes the strategy. Peripheral Not Visible Ocular motor and manual motor processing run independently. Interrupt when a blip becomes active. Peripheral Visible Maximizes human parallel processing that is built into the EPIC cognitive architecture Peripheral Not Visible Ocular Motor Preceding Movement (seconds) Time Preceeding Independent Eye and Hand Strategy Classification Time (seconds) Peripheral Not Visible Independent Subtask Strategy Peripheral Not Visible Peripheral Not Visible Red/Green YellowRed/Green Yellow Independent Subtask Strategy Peripheral Not Visible Time Preceeding Movement (seconds) Movements to Classify a Blip 8

13 Cognitive strategies in multitasking Skilled multitasking may involve running the ocular-motor and manual-motor processes independently and in parallel. There may not be independent strategies or mechanisms that actively coordinate between two task strategies (as in Kieras, Ballas, & Meyer, 00; Salvucci and Taatgen, 008). Eye movement data can help reveal the strategic interleaving of perceptual and motor processes in a timecritical dual task. Four Projects Exploring Cognitive Strategies for. Visual search of pull-down menus.. Visual search of hierarchical displays.. Multimodal multitasking.. Individualized strategies for multitasking Individualized cognitive strategies for multitasking Now we look for individual differences in the cognitive strategies that participants used in the multimodal dual task. ll participants use this overall strategy. But what, for example, are the subtle differences in how individuals decide how and when to make these transitions. Yunfeng Zhang s 0 dissertation work. Now at IBM TJ Watson. Look at tracking target tracking cursor color move joystick Key: Task switching Ocular Motor Interrupt when a blip becomes active. Resume after the active blip's features are encoded. Manual Motor Interrupt when a blip is ready to be keyed in. Resume when the blip is classified. Subtask delegation Classification Look at blip blip features Classification keypress Perceptual information

14 Most blips were classified with two to three eye movements Radar Classification Display. Eyes to target Blip search. Eyes to radar Pre-radar. Eyes to tracking Post-radar Display Four Strategy Dimensions. -to-radar-priority. When to move the eyes to the radar display after knowing a blip change color. Immediate-Eyes-to-Blip (IEB). Move immediately. Track-then-Eyes-to-Blip (TEB). until the cursor is green.. -to-radar-sound. When to move the eyes to the radar display after an auditory cue. Eyes-to-ll-Sounds (ES). Move for all sounds. Eyes-to-Color-change-Sounds (ECS). Move only for color-change sounds.. -to-radar-location. In the peripheral-not-visible conditions, where to put the eyes in the radar display when switching to the classification task. Look-Window-Center (LWC). Go to center. Look-prior-Blip-Location (LBL). Go to a black blip recalled from a previous visit.. Radar-to--Priority. What to do with the hands after the model acquired the visual features of a yellow blip. Keypad-Then-Joystick (KTJ). lways key-in the classification response before doing any tracking. Keypad-If-Green (KIG). If the tracking cursor is not green, do some tracking until it is green, and then key in. Joystick-Then-Keypad (JTK). lways do some tracking before keying in the classification response. Explored all possible paths through the master strategy. T-R-Priority IEB TEB T-R-Sound ES ECS T-R-Location LWC LBL R-T-Priority KTJ JTK KIG Ran the strategies through a Parallelized Cognitive Modeling System Two key components: Model Spawner: utomatic model generation. Job Scheduler: Parallel model execution. Parameter Space Parameter X: Range from 00 to 00 ms, sampled at every 00 Parameter Y: Model X=00 Use strategy The Cluster Model n + Model Model Basic Model Parameter X=? Strategy =? Model Spawner Model X=00 Use strategy Job Scheduler EPIC + + Submit to cluster EPIC EPIC Strategy Space Dimension : Strategy Strategy Dimension B: Model n X=00 Use strategy CPU Core CPU Core CPU Core Key: File Program Input/Output ssign 6

15 The system initially produces a cloud of predictions smaller than the cloud of individualized performance data. Used EPIC s default encoding time parameters. Peripheral Not Visible Peripheral Visible RMS Error (pixels) Classification Time (seconds) 7 Human: Model: Peripheral Not Visible Peripheral Visible RMS Error (pixels) Classification Time (seconds) nd so the model parameters are calibrated to individual data., the worst performer. Peripheral Not Visible Peripheral Visible RMS Error (pixels) Classification Time (seconds) 8 Human: Model: fter the model s parameters are calibrated for s encoding time for text and speed-and-direction, and keystroke execution time. Peripheral Not Visible Peripheral Visible RMS Error (pixels) Classification Time (seconds) nd so the model parameters are calibrated to individual data., the best performer. Peripheral Not Visible Peripheral Visible RMS Error (pixels) Classification Time (seconds) fter the model s parameters are calibrated for s encoding time for text and speed-and-direction, and keystroke execution time. 9 Human: Model: 60 Encode Blip Select Response Key-in Response Find Blip Notice blip change color Classification Stages R-T-Priority Hostility Encoding Time (HET) T-R-Priority T-R-Sound T-R-Location Eyes on Eyes on Radar Eyes on Time Strategic Dimension and Free Parameter figure out which substrategy each participant is using, it is determined which strategic dimension contributes to each time period in the eye movement data.

16 The tracking-to-radar-sound and tracking-to-radarpriority strategic dimension contribute to eyes-ontracking time. Measures Classification Stages Eyes on Eyes on Radar Eyes on Pre-radar Notice blip change color Blip search Find Blip Encode Blip Select Response Post-radar Key-in Response Compare the prediction to the observed within that one period in the eye movement data Pre radar duration (ms) Model: Human: T-R-Priority IEB TEB T-R-Sound ES ECS Strategy and Free Paramter T-R-Sound T-R-Priority T-R-Location Hostility Encoding R-T-Priority Time (HET) 6 Time Sound: On Off On Peripheral Visible: No Yes Yes For the -to-radar-sound dimension, Eyes-to- Color-change-Sounds (triangles) fits the sound-on peripheral-not-visible condition best. 6 6 Continue this process for each strategic dimension to determine the individualized strategy for each participant p performers make some strategic decisions that are different from the bottom performers. Participants p Performers Bottom Performers Strategy Dimensions interact RMSD HET T-R- T-R- T-R- R-T- (ms) EM CT TE Priority Sound Location Priority (ms) (ms) (pixels) IEB ES LBL KTJ P0 TEB ECS LBL KIG P07 IEB ECS LBL KIG P TEB ECS LBL JTK TEB ECS LBL KIG P7 { IEB ECS LBL KIG P0 { TEB ECS LBL JTK TEB ECS LBL KIG P09 TEB ECS LBL JTK P IEB ECS LBL KIG P6 TEB ECS LBL JTK TEB ECS LBL JTK Individualized cognitive strategies for multitasking Multitasking requires microstrategy selection, coordination, and execution. Individualized multitasking performance can be explained by individualized substrategy selection. Multitasking is a skill, but not a general skill. It is a skill that needs to be learned by each individual person, for each pair of tasks. 6

17 Once you think in terms of cognitive strategies, you think differently about interface design When you design an interface, you are designing a cognitive strategy. Students specify a user interface as a state transition diagram. Start "Main" menu is active B "Entering Main Menu" "Set Day" menu is active. B Main Menu: "Set Day" is highlighted "Set Day" Day of week menu item Sunday is highlighted "Sunday" C C Main Menu: "Set Hour" is highlighted "Set Hour" "Entering 'Set Day' menu. Current setting is <current setting>" Main Menu: "Set Minute" is highlighted "Set Minute" Note: The "Set Hour" and "Set Minute" states are identical to "Set Day", but of course with hours and 60 minutes. Day of week menu item Monday is highlighted "Monday" "You selected <selected day>" Day of week menu item Tuesday is highlighted "Tuesday" Main Menu: "Exit Program" is highlighted "Exit Program" B B "Exiting Program" End I have the students in my user interface classes think about interfaces in terms of cognitive strategies. 6 The compactness of the instructions relate to the compactness of the cognitive strategy, and ease of use. Key to this diagram "Text displayed auditorily when making a transition." Transition caused by pressing the "Left" key Name of the state Text displayed auditorily when entering state Transition caused by pressing the "Select" key Transition caused by pressing the "Right" key Key Mappings Left (J) Right (K) Select (Space) 66 Cognitive strategies could have helped to predict the customer complaints after a recent software update. The new version requires more eye and hand movements to do the same tasks. Visual search is best characterized as cognitive strategies moving the eyes to optimize the availability of the visual features that are needed to do a task. Because eye movements are always directed by the active cognitive process, an explanation of the eye movements must rely on an understanding of the controlling cognitive strategy. Russo (978) n obsession with covert shifts of attention has held back the study of visual search for decades. OmniFocus Version OmniFocus Version 67 68

18 Visual search is best characterized as cognitive strategies moving the eyes to optimize the availability of the visual features that are needed to do a task. Visual search is best characterized as cognitive strategies moving the eyes to optimize the availability of the visual features that are needed to do a task. EPIC s new availability functions will replace its fixed-diameter foveal regions. Probability that feature will be perceived Color Size Shape Eccentricity (in degrees of visual angle) There is little or no evidence that, when the eyes are permitted to move freely, covert visual attention moves independently of the eyes in a visual search task. Findlay & Gilchrist (00) 69 Findlay & Gilchrist (00) 70 Future Work: Semi-automated cognitive strategy generation Future Work: Semi-automated cognitive strategy generation 0. Do NRL dual task. Plan 0: Time-share between subtasks and... - Visually locate target and cursor. V. Do tracking task.. - Determine if target is currently being accurately tracked. V M M V H H Plan : Do subtask. If required after subtask, do subtask. Loop. Plan.: Do each subtask, in order. V M H H V. - djust tracking cursor... - Visually examine target and cursor. V M H H Plan.: Do each subtask, in order... - Determine error between the target and cursor... - Determine manual movement needed to reduce error. V V M H H. Do tactical task. V Plan : Hold on subtask until one or more blips are ready to be classified. Then, until all blips are classified, loop through subtasks and. Then, go back to holding on subtask.. - Determine that one or more blips are ready to be classified. V V V M H H V M H H.. - Execute manual H movement. V M H. - Select a blip to be classified. V V V M H H Task Condition Peripheral visible, Sound on or off Peripheral not visible, Sound on V M H H V. - Classify the blip. Plan.: Do each subtask, in order... Get visual features needed to classify blip... Determine blip's classification. M V H H V udition Vision - foveal.. Key-in response. V H M H Key Human Info. Processing Required in Leaf Nodes Vision - peripheral Memory retrieval.. Confirm correct response. M V Hands - with eyes Hands - eyes-free V V H H V H M H 7 udition Foveal V Vision Peripheral V Vision Memory Retrieval M Hands H with Eyes Hands H Eyes-Free Get blip features... Tactical time Overlapped processing Examine target and cursor..... Determine classification... Key-in response... Determ. error... Overlap. The hands key-in a tactical task response after the eyes move to tracking. Waiting for blip Determine cursor error and mvmt. needed. Overlapped processing Tactical djust cursor.... Select blip. Overlap. fter an auditory signal is received, a little more tracking is completed, and then a switch to the tactical. Get Blip features... Blip Overlapped processing Select Blip. Get features.... Blip.. Determine classification. Key-in response... Confirm response... Determi. classific... Overlap. Two blips are classified in an overlapped, "pipelined" manner. 7

19 Cognitive strategy are important for understanding and predicting aspects of human-computer interaction deny the importance of cognitive strategies in such a model would seem to lead to a slippery slope back to behaviorism. nd yet, cognitive strategies are not included in most HCI models of menu search. (Sears 99; Findlater 00; Cockburn 009). Most models are regression fits of specific sets of data. arrive at a useful engineering model, I believe we need to unpack the regression functions and identify the architectural and strategic components that generate the data. Cognitive strategies are a critical component of computational cognitive models of HCI tasks. 7