An Investigation of the Effect of Latency on the Operator's Trust and Performance for Manual Multi-robot Teleoperated Tasks

Transcription

1 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 390 An Investigation of the Effect of Latency on the Operator's Trust and Performance for Manual Multi-robot Teleoperated Tasks Hunter Rogers 1, Amro Khasawneh 1, Jeffery Bertrand 2 & Kapil Chalil Madathil 1 1 Department of Industrial Engineering, Clemson University 2 Department of Computer Science, Clemson University Latency is an important factor when conducting teleoperated missions. This study investigates the effects of latency on a set of dependent variables: performance (measured by time and number of errors), subjective workload, trust, and usability. These measures were tested in a simulated search-and-rescue mission over two levels of two independent variables. One independent variable was the number of robots one or two (within-subject), and the other independent variable was latency simulations with and without latency (between-subject.) The significant effect of the independent variables on the dependent variables were checked using repeated measure two-way ANOVA with a confidence level of 95%. The data determined any significant effects that latency and/or the number of robots had on such factors as errors, dependability, reliability, harmful outcomes, temporal demand, and frustration. Copyright 2017 by Human Factors and Ergonomics Society. DOI / INTRODUCTION Teleoperated robotics is an increasingly significant area of research in human robot interaction. The use of teleoperated robotics for distance applications may be necessary for logistical reasons or safety as in cases such as military, search- and-rescue, and space exploration. Critical missions conducted through teleoperations are becoming a reality and Human Factors professionals are researching different conditions of teleoperations and their effects on the human operator. Human operators still play a large role in these types of missions even with various levels of autonomy (Chen, Haas, & Barnes, 2007). One key factor in teleoperated robotics is time delay, or latency, in the system s input response and visual feedback. Physical transmission distances and limited available bandwidth can cause latency in a control and interface (Luck, McDermott, Allender, & Russell, 2006). One study found that latency, or lag, times over one second can significantly impact the time it takes to complete a task using a simulated teleoperated robot (Lane et al., 2002). Another study using a driving simulator with visual time delays showed a significant detriment to a person's well-being and performance through physiological and performance measures (Frank, Casali, & Wierwille, 1988). Luck and colleagues (2006) determined, when latency increased, the time to complete a teleoperated robotic task increased and the number of errors increased (Luck et al., 2006). Workload and trust are critical to the usability of a teleoperated robotic interface and to the well-being of the operator. Trust in human robot interactions has been difficult to measure. The subjective nature of trust leads to subjective measures as the primary way to quantify a human s trust in a robotic system outside of the binary measure of system utilization. Given the frequency of lag time in teleoperated tasks, it is imperative to research the effects of latency on workload and trust and develop design improvements to neutralize the degradative effects of latency (Chen, Barnes, & Harper-Sciarini, 2011; Hoffman, Johnson, Bradshaw, & Underbrink, 2013). Applications of teleoperated robotics are extensive. Both military and search-and-rescue fields want to remove the operator from targeted environments for safety reasons. However, to do so, an operator is still required to complete critical tasks. As the U.S. military transitions to singleoperator multi-vehicle control scenarios, research into the ability of the operator to maintain the performance, workload, situational awareness, and trust needed to interact with telerobotics in latency situations will influence design considerations for future missions (Riley, Strater, Chappell, Connors, & Endsley, 2010). Space exploration, such as the Mars Rover, must complete tasks with extreme transmission ranges that subject operators to large amounts of latency. Teleoperated healthcare also has the potential to allow skilled physicians and surgeons to treat remote locations within the body, or inaccessible regions due to disease outbreaks. As there is a limited amount of recent research in human telerobotic interaction, specifically on the effects of latency in User-To-Robot (U2R) signals on workload and task performance in multiple robot situations, this study examines workload through subjective measures, task performance, and subjective system trust. Specifically, the purpose of this study is to answer the following questions: RQ1: How does increased latency affect task performance? RQ2: How does increased latency affect subjective workload, specifically frustration levels? RQ3: How does increased latency affect trust in the robotic system? RQ4: What are the effects of controlling a second robot on performance and workload? METHOD Participants A sample of 40 students (21 females/19 males, age = 20-34, M= , SD=2.96) participated in the study. 97.5% of participants reported using a computer daily and 70% reported playing video games weekly; however, only 15% of participants had experience using a joystick and none had previous experience with the research platform. More information about participant demographics can be found in

2 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 391 Table 1. All the participants were required to read and confirm their understanding of the informed consent form before starting the study. Each participant was given a $10 Amazon gift card as compensation. The study has been approved by Clemson University s Institutional Review Board. Table 1: Participant Demographics Variable Number % Gender Male Female Education Some College year Degree Masters Degree 0 0 Doctoral Degree 6 15 Computer Usage Daily Weekly Video Game Usage Weekly None at all Joy Stick Usage At least once 6 15 None at all Apparatus A desktop computer with a 22-inch screen equipped with a Logitech Extreme 3D PRO joystick and Tobii X60 mobile eye tracker were used to conduct the study and collect data. Hypotheses H1: Task performance (measured by the amount of time taken to complete the task and the number of errors made) will decrease as latency increases and increase as the number of robots increases. H2: Perceived workload, specifically frustration, will increase as latency increases and increase as the number of robots increases. H3: Trust in the system will decrease as latency increases. Experimental Design The study used a mixed experimental design. The between-subjects variable was the latency level and the within-subjects variable was the number of robots. The participants were randomly assigned to four different conditions: 1. No latency, controlling one robot initially 2. No latency, controlling two robots initially 3. With latency, controlling one robot initially 4. With latency controlling two robots initially Using a joystick to control one or two robots, each participant was asked to complete as quickly as possible the simulated rescue of 10 victims stuck in a building after an earthquake. An overhead map that revealed the layout of the building as the operators explored the environment was provided to help participants identify parts of the building. A real-time trust rating appeared every two minutes asking the participant to rate their trust in the system. Trust, in this experiment, was defined as the operators trust in the entirety of the system and further explained as the operators trust in the robot s ability to complete the assigned task, in the accuracy information displayed on their screen, and in the accuracy of the joystick control. Screenshots of the tworobot and one-robot simulations with the trust ratings are shown in Figures 1 and 2, respectively. Figure 1: Finding a victim in a two-robot simulation Independent variables. The two independent variables for this study are the latency level and the number of robots. 1. Latency (between-subjects): To study whether latency in the U2R signal causes degradative effects in performance, increases subjective workload and frustration, and decreases trust in the system, the participants were exposed to one of two levels of latency during the simulation: 0 milliseconds (no latency) and 500 milliseconds (with latency.) 2. The number of robots (within-subjects): To investigate whether controlling a second robot affects the performance and workload, participants controlled one and two robots at different settings. Figure 1 represents the settings for controlling two robots and Figure 2 shows the settings for controlling one robot Dependent variables. Twenty-six dependent variables were measured and analyzed, including time to complete the task, number of errors made during the completion of the task (Boduroglu, Minear, & Shah, 2007), real-time trust in the system, overall trust in the system (Jian, Bisantz, & Drury, 2000), the IBM Computer System Usability Questionnaire scale (IBM-CSUQ) (Lewis, 1995), and the NASA Task Load Index test (NASA-TLX) (Hart & Staveland, 1988). Although workload may be effected by trust, these variables were measured separately. Procedure. Upon arrival, one of the researchers provided the participant with a brief overview of the study and asked him/her to read an informed consent form and fill out a demographic survey. Then, the researcher explained the task to the participant and took him/her through a training session to familiarize the participant with the controls and the simulator. The training session was conducted using the same latency as the trials that the participant would complete and had the same settings as the measured study, but with less victims to rescue (only three people.) Further, step-by-step instructions on how to use the system were provided at the bottom of the screen to guide the participant through the session. Once the eye tracker was

3 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 392 calibrated, the participant then performed one of the experimental tasks described in the experimental design section. Subsequently, the participant was asked to fill out an overall trust questionnaire, followed by the IBM-CSUQ scale and NASA- TLX indices. The procedure was then repeated for the second condition. The overall procedure took less than one hour. Figure 2: Real-time trust rating in one-robot simulation RESULTS The significant effect of the independent variables on the dependent variables were checked using a repeated measure two-way ANOVA with a confidence level of 95%. To do so, SPSS 24 was used. There were some outliers based on studentized deleted residual and Cook s distance. All the outliers were removed from the analysis. The data was normally distributed as assessed by skewness and kurtosis values. The assumption of sphericity was not met for the two-way interaction in all the dependent variables as indicated by Mauchly s test. Two-Robot Strategies On average, users found the second robot to be helpful, rated 2.7 on a scale of 1 to 5 (1 being extremely helpful, 5 being not helpful at all.) When participants were asked to describe any strategies used to accomplish the mission, 21 of the 40 participants mentioned using the second robot and rated the helpfulness at 2.66 on average. Seventeen cited using the second robot to split the work and distance, and three participants explained that they only used the second robot if the first robot got stuck, or they became frustrated. Performance Measures Time Taken. Simulation time was measured by the system in seconds from the time the user started controlling the robot to the time the last (10th) victim was found, excluding any time taken to complete the real-time trust rating. The analysis indicated no significant effect of either independent variable on the time to complete the task, F (1, 38) = and F (1, 38) = 1.877, p > 0.05, for number of robots and latency, respectively. Errors. Errors were measured by the number of times the robot stopped due to an obstacle (wall, door frame, desk, etc.) that blocked the robot s view. The results revealed a main effect of latency on the number of errors, F (1, 36) = 27.81, p < 0.05, as shown in Figure 3. The number of errors made by the non-latency group (M= 7.667, SD = 1.422) were significantly less than the number of errors made by the latency group (M= 18, SD = ) Subjective Measures Overall Trust. Different subjective measures were used to measure the participant s trust in the system: deceptiveness, underhandedness, suspicion, wariness, harmful outcomes (user s assessment of if the robot s actions may have harmful outcomes), confidence, security, integrity, dependability, reliability, familiarity, overall trust, and average real-time trust. Of these variables, there was a significant two-way interaction between latency level and number of robots on dependability, F (1, 36) = 4.4, p <0.05, as shown in Figure 5. The simple main effects showed that dependability was not significantly different for the nonlatency group (M= 5.158, SD = 0.31) compared to the latency group (M= 4.789, SD = 0.31) when using one robot, F (1, 36) = 0.707, p > However, dependability was statistically significantly greater in the non-latency group (M= 5.53, SD = 1.47) compared to the latency group (M= 4.32, SD = 1.33) when using two robots, F (1,36) = 7.07, p < There was also a two-way interaction between latency and the number of robots on reliability, F (1, 38) = 6.8, p<0.05, shown in Figure 6. However, the simple main effects showed that the simple effects are not statistically significantly different from zero. This outcome indicates that the slopes have different signs (one is positive, and the other is negative); therefore, they are different from each other but each one of them is not different from zero. Based on Figure 6, reliability showed no difference between the non-latency group (M= 4.95, SD =1.7) and the latency group (M= 4.95, SD = 1.43) when using one robot. However, reliability was greater for the non-latency group (M=5.4, SD = 0.355) compared to the latency group (M= 4.4, SD = 0.355) when using two robots. There was an interaction effect of latency and the number of robots on harmful outcomes, F (1, 37) = 6.3, p<0.05, shown in Figure 7. The simple main effects showed that harmful outcomes were not significantly different between the non-latency group (M= 1.25, SD = 0.087) and the latency group (M=1.105, SD = 0.089) when using one robot, F (1,37) = 1.363, p > However, harmful outcomes were significantly lower for the non-latency group (M= 1.05, SD = 0.224) compared to the latency group (M= 1.42, SD = 0.77) when using two robots, F (1,37) = 4.3, p < Workload. There was an interaction effect of latency and the number of robots on temporal demand. However, the simple main effects test denoted that the simple effects were not statistically significantly different from zero; therefore, the

4 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 393 Figure 3: Effect of Latency on Number of Errors Figure 4: Effect of Number of Robots on Frustration Figure 5: Interaction Plot of Dependability Figure 6: Interaction Plot of Reliability Figure 7: Interaction Plot of Harmful Outcomes Figure 8: Interaction Plot of Temporal Demand slopes have different signs. Based on Figure 8, temporal demand showed no difference between the non-latency group (M= , SD =1.886) and the latency group (M= 13.85, SD = 1.838) when using one robot. However, it was greater for the non-latency group (M=15.139, SD = 1.584) compared to the latency group (M= 10.95, SD = 1.544) when using two robots. There was a main effect of the number of robots on frustration level, F (1, 37) = 4.16, p < 0.05, shown in Figure 4. When participants controlled one robot (M= 8.54, SD = 6.9), the frustration level was less than when they controlled two robots (M= 11.13, SD = 9.) There was neither main nor interaction effects of the independent variables on the remainder of the dependent variables. DISCUSSION In this research, there were no significant effects on any of the independent variables when measuring the time taken to complete the task, likely due to the variability in the data. The variability could be attributed to differences in strategies or memory limitations some participants relied on their memory to assess what rooms they had checked rather than the map, given an observed tendency of participants to go in many rooms more than once in many participant s trials. In comparing latency conditions, when using one robot, most dependent variables showed no significant effect except on the number of errors. This effect on the number of errors may be attributed to overcorrection commonly seen in lag between user input control and visual feedback, causing the controlled robot to get stuck on walls or obstacles. However, when participants used two robots, the effects of latency start to appear in other dependent variables: dependability, reliability, harmful outcomes, and temporal demand. Participants reported that the system was less reliable and dependable and even felt there was more time pressure (temporal demand) although the task took participants no additional time on average. In addition, the participants rated the possibility of harmful outcomes higher in the two-robot condition. This observation confirms a part of H3: increased dependability and reliability are linked to increased trust, and increased possibility of harmful outcomes is linked to mistrust. In other words, the participants inherently trusted the system less as latency increased when using two robots. This observation also confirms a portion of H2: increased time pressure is linked to increased workload, and increased latency and the number of robots increases workload. The number of robots had a significant effect on the subjective rating of frustration. Participants reported lower frustration levels using one robot than using two robots, confirming a portion of H2. This outcome is possibly due to higher working memory demands as participants develop strategies to use the secondary robot and maintain situational awareness of a system of two robots. Another possibility of increased frustration is task switching. The robots only move through manual control and no autonomy was developed for the robots. Participants had to switch between robots to utilize both robots rather than allocating certain tasks or

5 Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 394 areas to one robot while simultaneously working with the second robot. The participants were required to switch between robots to control both, which may have increased the burden on working memory. Elements of trust were measured with a survey of validated statements that quantified trust. Some of these elements were found to be significant, but the average realtime trust measurement did not reveal a significant difference between latency conditions. This outcome could be attributed to the subjective nature of trust. Perceived robustness and reliability of the system are factors of trust, but individual differences, such as culture and personality, also affect trust (Endsley, 2016; Sadrfaridpour et al. 2016). Based on specific elements of trust being significant, we find evidence to support H3. Given the current models of trust and that generalized trust is subjective and effected by personal elements that differ from person to person, there is reason to believe that specifically stated, validated elements of trust may better quantify overall trust rather than general statements of trust. Limitations One major limitation of this study was sample size. After completing the statistical analysis of the data, some of our dependent variables that we thought would be affected by the level of latency did not show a statistically significant effect. A power analysis was conducted and it was determined that our sample size needed to be larger than 150. Based on these results, a future study will be conducted with a much larger sample size to determine if significant interaction can be found. This study also did not focus on creating a high face validity simulated environment and, as the task was not controlling robots in real-time, there may be a limitation in applying this research to real-time teleoperation of robots. Conclusions Findings of this study show promise of significant effects of latency on performance, trust, and workload. Continued study will determine if significant effects are seen on other measures of workload and trust with larger sample sizes. Future studies will investigate levels of automation (LOA) in this simulation task, determining if increased LOA will mitigate the effects of latency on task performance, trust, and workload without compromising necessary situational awareness. Future research will also consider how system design, specifically the effects of sonifcation, force feedback, or interface design, will impact task performance, trust, workload, and situational awareness in latency conditions. REFERENCES Boduroglu, A., Minear, M., & Shah, P. (2007). Working Memory. In Handbook of Applied Cognition (pp ). John Wiley & Sons Ltd. Chalil Madathil, K., Koikkara, R., Gramopadhye, A. K., & Greenstein, J. S. (2011, September). An empirical study of the usability of consenting systems: ipad, Touchscreen and paper-based systems. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 55, No. 1, pp ). Sage CA: Los Angeles, CA: SAGE Publications. Chen, J. Y. C., Barnes, M. J., & Harper-Sciarini, M. (2011). Supervisory Control of Multiple Robots: Human-Performance Issues and User- Interface Design. IEEE Transactions on Systems, Man and Cybernetics. Part C, Applications and Reviews: A Publication of the IEEE Systems, Man, and Cybernetics Society, 41(4), Chen, J. Y. C., Haas, E. C., & Barnes, M. J. (2007). Human Performance Issues and User Interface Design for Teleoperated Robots. IEEE Transactions on Systems, Man and Cybernetics. Part C, Applications and Reviews: A Publication of the IEEE Systems, Man, and Cybernetics Society, 37(6), Endsley, M. R. (2016). From Here to Autonomy. Human Factors. Frank, L. H., Casali, J. G., & Wierwille, W. W. (1988). Effects of visual display and motion system delays on operator performance and uneasiness in a driving simulator. Human Factors, 30(2), Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Peter A. Hancock and Najmedin Meshkati (Ed.), Advances in Psychology (Vol. Volume 52, pp ). North- Holland. Hoff, K. A., & Bashir, M. (2015). Trust in Automation. Human Factors, 57(3), Hoffman, R. R., Johnson, M., Bradshaw, J. M., & Underbrink, A. (2013). Trust in Automation. IEEE Intelligent Systems, 28(1), Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an Empirically Determined Scale of Trust in Automated Systems. International Journal of Cognitive Ergonomics, 4(1), Lane, J. C., Carignan, C. R., Sullivan, B. R., Akin, D. L., Hunt, T., & Cohen, R. (2002). Effects of time delay on telerobotic control of neutral buoyancy vehicles. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Vol. 3, pp ). Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human Computer Interaction, 7(1), Luck, J. P., McDermott, P. L., Allender, L., & Russell, D. C. (2006). An investigation of real world control of robotic assets under communication latency. In Proceedings of the 1st ACM SIGCHI conference on Human-robot interaction (pp ). ACM. Madathil, K. C., & Greenstein, J. S. (2017). An investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testing. Applied Ergonomics. Riley, J. M., Strater, L. D., Chappell, S. L., Connors, E. S., & Endsley, M. R. (2010). Situation awareness in human-robot interaction: Challenges and user interface requirements. In M. Barnes & F. Jentsch (Eds.), Human-Robot Interactions in Future Military Operations (pp ). Ashgate Publishing. Sadrfaridpour, B., Saeidi, H., Burke, J., Madathil, K., & Wang, Y. (2016). Modeling and control of trust in human-robot collaborative manufacturing. In Robust Intelligence and Trust in Autonomous Systems (pp ). Springer U