Hand Eye Coordination Patterns in Target Selection

Transcription

1 Hand Eye Coordination Patterns in Target Selection Barton A. Smith, Janet Ho, Wendy Ark, and Shumin Zhai IBM Almaden Research Center 65 Harry Road San Jose, CA 9512 USA {basmith, ABSTRACT In this paper, we describe the use of tracking and trajectory analysis in the testing of the performance of input devices for control in Graphical User Interfaces (GUIs). By closely studying the behavior of test subjects performing pointing tasks, we can gain a more detailed understanding of the device design factors that may influence the overall performance with these devices. Our Results show there are many patterns of hand eye coordination at the computer interface which differ from patterns found in direct hand pointing at physical targets (Byrne, Anderson, Douglass, & Matessa, 1999). Keywords Eye tracking, gaze, hand eye coordination, pointing, target selection, mouse, touchpad, pointing stick, motor control 1. INTRODUCTION Human computer interface research has traditionally focused on performance. A typical topic of such a nature is computer input. Input devices and techniques are usually tested against a set of standard tasks in which user s performance on task completion time and error rate is measured and analyzed (e.g. Card, English, & Burr, 1978). The results of the performance analysis serve as the basis for refinement and redesign of the devices and techniques. However, observations at the performance level often overlook important information on how users actually accomplish the task, which may offer additional insights toward a better understanding of the interaction process and design solutions. As the field matures, process oriented research, should begin to contribute to the understanding of interaction. In human motor control research, the study of the micro-structure has served similar purposes (Jagacinski, Repperger, Moran, Ward, & Class, 198). In order to understand the usability of various 6 degree-offreedom (DOF) devices, Zhai and Milgram (Zhai, 1998) recently studied both the performance and the trajectory of 6 DOF manipulations of 3D objects. Their trajectory analysis revealed critical differences between devices in terms of coordination, which would not be found in time performance data alone. In conjunction with trajectory analysis, Eye-tracking provides a comprehensive approach to studying interaction processes. In the field of HCI, eye tracking has helped to improve the understanding of how users search and select menu items (Card, 1982), (Aaltonen, Hyrskykari, & Raiha, 1998), (Byrne et al., 1999). Eye tracking has also been used in studying human pointing tasks with hands in the physical world. Given that most motor control movement is either initiated or guided by perception, it is necessary to understand how relates to hand movement. For example, Helsen and colleagues studied the temporal and spatial coupling of gaze and hand movement (Helsen, Elliott, Starkes, & Ricker, 1998). In a reciprocal pointing task with two fixed targets, they found a rather invariant patterns of hand eye movement relationship: tended be initiated 7 ms earlier than hand movement; typically makes two saccades to land on target and the first saccade tended to undershoot. The pattern of task termination was also very consistent: stabilizes on target at 5% of the total hand response time. Pointing on a computing screen with an input device may or may not follow the patterns found in pointing with the hand to physical targets. There are many reasons for different behavior, due to the various disparities between the hand motion and the motion (Wang & MacKenzie, 1999). For examples, direct hand pointing is carried out with proprioceptive feedback of hand position in the human arm. Pointing at graphical objects on a screen is carried out with a, which does not have a direct, absolute mapping with hand motion. This means that the user may have to sample the location with gaze in the course of a pointing trial, unless the motion can be perceived by peripheral vision. The mapping between motion and input device is often a complex transfer function, which may further increase the complexity of the hand eye relationship in target acquisition tasks with a computer. In the case of computer mice, most of them are power mice with non-linear acceleration schemes. More precisely, the control gain in a power mouse is not a constant, but depends on the speed of the mouse motion. Faster movement of the mouse results in higher control gain. In the case of a pointing stick such as Trackpoint 1, the input force is mapped onto velocity by a complex transfer function (Rutledge and Selker), with various plateaus to provide a speed easier for the to follow. Detailed study of gaze pattern is surely useful for further refining the transfer functions in these devices. 1 TrackPoint is a trademark of the International Business Machines Corporation.

2 In the case of a small touchpad often seen in laptop computers, multiple strokes often have to be made in order to move the to a distant target. Does this mean the user has to gaze at the in order to make each stroke? In summary, understanding the eye hand relationship serves as an important foundation for understanding and designing input methods. Study has shown invariant hand-eye coordination patterns in direct hand pointing tasks (Helsen et al). The disparities between hand and motion on GUI interface suggest possibly much more complex hand eye behavior in computer target acquisition tasks, such as occasional gaze switch between and target, or gaze focus on. This study makes an initial attempt to test and understand hand eye coordination patterns at the computer interface. 2. METHODS 2.1 Participants 24 volunteers participated in the experiment. All participants had normal or corrected to normal vision. All were right-handed and were experienced computer users in the Windows platform with at least three years of continuous usage. Half of the participants had experience with the pointing stick. None of the participants had experience with the touchpad. 2.2 Experimental Design The participants were required to perform two different tasks with three input devices. The three input devices in question were mouse, touchpad, and pointing stick. Each participant switched to another input device after performing the two different tasks. In total, each participant performed six tasks. The order in which they used the input devices and the order of the two tasks were randomly counter-balanced. Task one was a reciprocal pointing task. A pair of identically sized circular targets was placed diametrically around the center of the computer monitor at specified distances and directions. Targets were presented with all possible combinations of the following: distance, radius, and angle. The center-to-center distances were, 4, and 6 pixels. The radii were 1, 2, and 3 pixels. The angles (from horizontal) were 45, -3,, 3, and 45 degrees. Each target pair was used for two trials, resulting in a total of 9 trials for this task. The participants were to look at the monitor screen and use the input device to point and alternately select by clicking on the presented target circles. Task two was a random pointing task. The participants pointed and clicked on a set of randomly distributed circular targets presented sequentially on the monitor screen. The targets had radii of 1, 2, and 3 pixels in random order. Each participant was also required to complete ninety trials of this task. Task 1 is most common in Fitts law based input device research. Task 2 is closer to what a user typically does on a computer screen by pointing. The key difference between the two types of tasks lies in the predictability of the target. For Task 1, the first click on a pair of targets started the actual data collection for that pair of targets. For the following two measured trials, the participant already knew where the next target was. For Task 2, the participant could not predict where a target would appear until the trial actually began. This difference may influence the hand-eye coordination pattern. Each subject received exactly the same set of targets. The target generation program used was a Java application called IDTest, available from The IDTest program ran on a Pentium-based 167 MHz computer. The targets were displayed at a resolution of 124 by 768 pixels by high colors (16 bits) on an IBM P21 monitor, using an ATI 3D PRO Turbo PC2TV video card. The viewable area of the screen was.365 m horizontal by.28 m vertical at a distance of.64 m from the eye. The screen refresh rate was 9 Hz. Three input devices were used in this experiment: an IBM mouse (model 12J3618), a Cirque SmartCat 2 touchpad, and an IBM TrackPoint pointing stick in a desktop keyboard. 2.3 Eye-tracking system Eye gaze position was tracked by an Applied Science Laboratories (ASL) Model 54 eye tracker unit. The tracking software ran on a Pentium MHz computer. This unit tracks gaze position by observing the position of the pupil and front surface reflection from a single eye. A chin rest was used to stabilize the participants viewing position and distance. A scan converter (Focus Enhancements TView Gold) was used to produce a combined video image signal of the targets displayed on the computer monitor and the eye position calculated by the eyetracker unit. This composite view was used by the experimenter to verify that the gaze tracking was working. Eye movement data were recorded every 6th of a second (6 Hz update rate) and averaged over every four data points through the ASL eye tracker interface program. The coordinates were recorded by IDTest. The eyetracker movement data were streamed into the P167 computer through a serial port and IDTest then combined eyetracker movement data and data in one single data file with respect to time. The eyetracking data obtained from the ASL eyetracker were converted into the same coordinate system as the coordinates. The calibration points on the ASL eyetracker and the stimulus machine were used to obtain the parameters for the conversion. 3. RESULTS We first look at the effect of device on overall pointing performance. Figure 1 shows the task completion time (the sum of all 9 trial completion times) for each device averaged over all tasks and subjects. Figure 1 shows that the device significantly affected performance time, F(2,44)=5.19, p<.1. Mean completion times were 12, 157, and 198 for the mouse, pointing stick and touchpad respectively. These results support previous findings for IBM TrackPoint device and touchpad performance times (Douglas, Kirkpatrick, and MacKenzie, 1999). We then searched for a constant invariant pattern that Helsen and colleagues found in direct hand pointing (Helsen et al 199?) through various statistical methods. However, after many attempts we could not find a strong central tendency in our data. Depending on the individual, many different hand eye coordination strategies were be used even within the same input device. 2 SmartCat is a trademark of Cirque Corporation.

3 Dummy 225 sentence is a significant pause, after beginning the trial by clicking on the initial target, before rapid motion toward the next target begins. (Seconds) Trackpoint Touchpad Mouse Figure 1: Mean performance Times by Device In order to examine the performance of the tasks in more detail, we examined patterns in relationship to and target locations by individual trials. Individual trial patterns as well as overall trends for a particular participant given a specific device and task were plotted. We found that unlike in the Helsen study, the hand-eye relationship was not very consistent across participants. Three different pointing behaviors appeared: eye gaze following the to the target, leading the to the target, and switching between the and the target until the target is reached. The first two were the most prevalent for these participants time (seconds) Figure 2: 'Cursor following' behavior Figure 2 is an example of the following the in Task 1 using the TrackPoint pointing stick. The graph shows the Euclidean distance of the and the to target. For this trial, the distance between target centers was 6 pixels and the target radius was 3 pixels. The trial began when the subject clicked on the target of the previous trial. There was little change in or position for the first.18 s. Then the began moving toward the target with the following. About.6 seconds was required to bring the within 5 pixels of the target center, at which time both the target and were within the foveal vision area. An additional.4 seconds was required to move the into the target and click. See Appendix A for an example of the aggregate data that shows this behavior for one participant. In Task 1, the subject knows in advance the position of the target for the next trial. Even so, there time (seconds) Figure 3: 'Target gaze' behavior Figure 3 is an example of the leading the to the target in Task 1 using the pointing stick. The target distance was 6 pixels and the target radius was 1 pixels. Both and position remained constant for the first.3 s of the trial, and then the moved rapidly to a position near the target. Essentially the person is focusing on the target rather than the. The followed to within pixels of the target in about.9 s, where it paused then went through and slightly beyond the target. An additional 1.8 s is required to bring the within the target and to click, during which time both and target are within the foveal vision area. See Appendix B for an example of the aggregate data, which show this behavior for one participant time (seconds) Figure 4: 'Switching' behavior Figure 4 is an example of switching from target to in Task 2 with a mouse. The target distance was 411 pixels and the target radius was 2 pixels. There was no motion, and thus no position recorded, for the first.17 s of the trial. When the was first moved, the was already near the target. This initial behavior is understandable since in Task 2, the new target appears in a random position. The subject must find the target visually before knowing in which direction the must be moved. The was moved rapidly to a position within less than pixels from the target center in about

4 .1 s. This is rapid initial movement phase is often seen with the mouse, due to the power mouse sensitivity acceleration. Then the was moved more slowly toward the target, during which time the gaze shifted back and forth from the target to the. Figures 5-7 are examples of one participant s and distance in relation to the target. These plots represent data from Task 1, only from the trials with distance of targets equal to 6 pixels. The figures 5-7 show the behavior using the pointing stick, mouse and touchpad, respectively. For this participant, the behaviors are very similar for all devices. There is a high concentration of points near the target before the draws closer to the target. For the touchpad plot, there are a couple trials when the arrives at the target in a very short period of time, but the high concentration of eye gaze points near the target are before the high concentration of points near the target. Figures 8-1 are examples of a different participant s and distance in relation to the target. Similar to the previous figures, these plots represent data from Task 1, only from the trials with target which are 6 pixels apart. Unlike the previous participant, who demonstrated similar behaviors across device, this participant seemed to change behavior. The pointing stick and touchpad behaviors are similar as both are exhibiting the target gaze behavior. However, for the mouse, the behavior cannot be categorized to either the target gaze behavior or the following behavior. In short, the hand eye coordination patterns were not determined by device alone. 4. DISCUSSIONS AND CONCLUSIONS In comparison to performance analysis, process-oriented study, especially eye tracking trajectory analysis, is much less mature and more complex. Many of the traditional techniques in studying performance, such as statistical variance analysis of

5 means, did not produce informative results in our study. When we averaged eye-tracking data in order to conduct variance analysis, we lost much of the information contained in the data. This may be partially due to the inadequacy of the method and partly due to the lack of one consistent pattern in the data, even for the same device and task parameters. Detailed, individual trial analysis and aggregate scatter plot analysis proved to me much more useful. We are planning to use alternative methods, such as data mining to further examine our data. When we started this project, we hoped to find consistent hand eye spatial and temporal relationship as Helsen and colleagues found in direct hand pointing tasks. They found that tended be initiated 7 ms earlier than hand movement; typically makes two saccades to land on target and the first saccade tended to undershot. Eye gaze stabilizes on target at 5% of the total hand response time. The result of our study showed the opposite: participants used a variety of hand eye coordination strategies in controlling to acquire targets. This is similar to the eye tracking study on menu selection by Byrne and colleague (Byrne et al, 1999) who predicted behavior with both ACT-R model and EPIC model, but neither model was confirmed or rejected. Eye movement pattern was more complex than either model could explain. In our current pointing task study, some participants used a strategy similar to direct hand pointing with the eye primarily on target and rarely on the. This means that without direct proprioceptive feedback of the position, which represents the hand action, some participants could use peripheral vision to monitor their motion. Alternatively, one can argue that these participants did not monitor their as all in the ballistic phase of pointing, as suggested by Woodworth s twophase theory of motor control (Woodworth, 1899). Woodworth s model suggests that the ballistic phase is triggered open loop behavior and only the second, phase near the target is guided closed-loop control behavior. Other participants used a strategy that is never reported in direct hand aiming tasks in the physical world. They continuously gazed at the, the visual representation of the physical hand, until the is in the vicinity of the target when both the and the target images are in the fovea. Yet other participants switched their attention back and forth between the and the target. This is also different from what the Woodworth model suggests in physical pointing. Overall, we found that participants used a variety of combinations of hand eye coordination patterns. This means that the design of input device algorithm should take all of these patterns into account. Designers cannot assume the one fixed hand-eye coordination pattern found in direct hand pointing. The trajectories alone also revealed some patterns different from those found in psychomotor studies where a bell curved velocity profile is often found. In our data, much faster speed is found at the onset of a trial, particularly when the input device was a power mouse. It is of both theoretical and practical importance to investigate if faster (than direct hand pointing) devices can be designed to take advantage of the non-linear transformation in input devices and various hand-eye coordination patterns found for computer target acquisition. 5. REFERENCES [1] Aaltonen, A., Hyrskykari, A., & Raiha, K.-J. (1998). 11 Spots, or How do users read menus? In Proceedings of CHI'98: ACM Conference on Human Factors in Computing Systems, Los Angeles, CA. [2] Byrne, M. D., Anderson, J. R., Douglass, S., & Matessa, M. (1999). Eye tracking the visual search of click-down menus. In Proceedings of CHI'99: ACM Conference on Human Factors in Computing Systems, Pittsburgh, PA. [3] Card, S. (1982, March 15-17, 1982). User perceptual mechanism in the search of computer command menus. In Proceedings of Human Factors in Computer Systems, Gaithersburg, Maryland. [4] Card, S. K., English, W. K., & Burr, B. J. (1978). Evaluation of mouse, rate controlled isometric joystick, step keys and text keys for text selection on a CRT. Ergonomics, 21, [5] Douglas, S. A., Kirkpatrick, A.E. and MacKenzie, I.S. Testing Pointing Device Performance and User Assessment with the ISO 9241, Part 9 Standard, in Proceedings of CHI '99 (Pittsburgh PA, May 1999) ACM Press, [6] Helsen, W. F., Elliott, D., Starkes, J. L., & Ricker, K. L. (1998). Temporal and spatial coupling of point of gaze and hand movement in aiming. Journal of Motor Behavior, 3(3), [7] Jagacinski, R. J., Repperger, D. W., Moran, M. S., Ward, S. L., & Class, B. (198). Fitts' law and the microstructure of rapid discrete movements. Journal of Experimental Psychology: Human Perception and Performance, 6(2), [8] Wang, Y., & MacKenzie, C. L. (1999). Effects of Orientation Disparity between haptic and graphic displays of objects in virtual environments., In Proceedings of INTERACT 99: IFIP International Conference on Human-Computer Interaction, Edinburgh, UK. [9] Woodworth, R. S. (1899). The accuracy of voluntary movement. The Psychological Review, Series of Monograph Supplements, 3(2 (Whole No. 13)), [1] Zhai, S., Mailgram, P. (1998). Quantifying coordination in multiple DOF movement and its application to evaluating 6 DOF input devices. In Proceedings of CHI'98: the ACM Conference on Human Factors in Computing Systems.

6 Appendix A All trials for a single subject, task, and device. 'Cursor following' behavior for Task 1 with mouse normalized time Appendix B All trials for a single subject, task, and device. 7 'Target gaze' strategy for Task 1 with TrackPoint distance (pixels) eye to target to target normalized time