UNIVERSITY OF CALGARY. Do In-Vehicle Systems Utilizing Voice-Recognition Technology Impact Driving Performance? A Systematic Review and Meta-Analysis

Transcription

1 UNIVERSITY OF CALGARY Do In-Vehicle Systems Utilizing Voice-Recognition Technology Impact Driving Performance? A Systematic Review and Meta-Analysis by Sarah Simmons A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE GRADUATE PROGRAM IN PSYCHOLOGY CALGARY, ALBERTA AUGUST, 2016 Sarah Simmons 2016

2 Abstract Newer model vehicles are often equipped with or capable of supporting hands-free systems that use voice-recognition technology. Although voice-recognition technology is viewed favourably among the public, it is not clear whether these systems should be considered safe alternatives to traditional handheld phones and visual-manual integrated systems. To answer this question, an exhaustive search was conducted to capture all experimental studies involving secondary tasks with voice-recognition systems where driving performance was measured. Meta-analyses for the performance measures of detection, reaction time, lateral control and longitudinal control were conducted with 43 studies meeting inclusion criteria. Some driving performance benefits were observed relative to visual-manual systems, but there were also considerable impairments relative to baseline driving. The results of the study indicate that voice-recognition systems, despite minimizing eyes-off-road time, have a distraction cost. Implications for driver education, voice-recognition system design and future research are also discussed. Keywords: driver distraction, driving performance, voice-recognition, speech-to-text ii

3 Acknowledgements I would like to express my sincere gratitude to my supervisor, Dr. Jeff Caird, for providing guidance, support and wisdom throughout my studies. I would also like to thank my committee members for their guidance, assistance and constructive feedback: Dr. Tom O Neill and Dr. Andrea Protzner of the Department of Psychology, and Dr. Piers Steel of the Department of Human Resources and Organizational Dynamics at the University of Calgary. A very special thanks to the library staff in the Interlibrary Loan Department within Libraries and Cultural Resources at the University of Calgary, who were instrumental in moving this project forward. Special thanks to Laura Koltutsky, liaison librarian for Psychology, Social Work and Sociology at the University of Calgary, who assisted me in developing a targeted list of electronic databases to search for studies. I would also like to thank all of the authors who provided additional study information and data upon request. I would also like to acknowledge Jasmine Mian and Katelyn Wylie, my CERL lab mates, as well as Justin Baers, for their advice and friendship. Finally, I would like to thank everyone who has provided encouragement and support throughout my studies, particularly my family and friends. I would also like to acknowledge my sister, Laura, for her helpful comments and feedback on this paper. iii

4 Table of Contents Abstract... ii Acknowledgements... iii Table of Contents... iv List of Tables... vi List of Figures and Illustrations... vii Introduction... 1 Method... 6 Eligibility Criteria... 6 Participants... 6 Independent variables... 6 Dependent variables... 6 Detection... 7 Reaction time... 7 Speed... 8 Headway... 8 Lateral position... 8 Study design... 9 Exclusion Criteria... 9 Information Sources Search Strategy Study Selection Data Collection Data Items Risk of Bias in Individual Studies Study Quality Data Synthesis Risk of Bias Across Studies Results Included Studies Meta-Analysis Detection Reaction time Standard deviation of lane position (SDLP) Mean deviation of lane position (MDLP) Headway Headway variability Speed Speed variability Meta-Regression Experimental setting Voice-recognition accuracy Year of publication iv

5 Publication Bias Risk of Bias Assessment Study Quality Discussion Theoretical Implications Practical Implications for Drivers Practical Implications for Designers & Manufacturers Limitations Future Directions Conclusions References Appendix A: DATABASE SEARCH Appendix B: SENSITIVITY ANALYSES Appendix C: MISSING VALUE ANALYSIS Appendix D: PRISMA CHECKLIST Appendix E: RISK OF BIAS ASSESSMENT Appendix F: STUDY QUALITY ASSESSMENT v

6 List of Tables Table 1 Overview of Included Studies Table 2 Results of Meta-Analysis Table 3 Summary of Effects Associated with Voice-Recognition Systems vi

7 List of Figures and Illustrations Figure 1 Study selection vii

8 Introduction Over the past ten years, fatalities associated with distracted driving have been increasing (Wilson & Stimpson, 2010). In 2013, 10% of fatal crashes, 18% of injury crashes, and 16% of all police-reported crashes involved driver distraction in the United States (NHTSA, 2015). As a countermeasure, safe driving legislation targeting handheld cell phone use has been implemented in almost all US states and all Canadian provinces (CCMTA, 2013; GHSA, 2015). Research shows that the dangers of distracted driving are not unappreciated by the general public, but awareness of the dangers associated with distracted driving does not necessarily translate into behavioural modifications among drivers (Hamilton, Arnold, & Tefft, 2013). The demand for communication and connectivity behind the wheel, in conjunction with the implementation of safe driving legislation targeting handheld mobile phones, has led to the adoption of hands-free technologies. Notably, hands-free technologies allow drivers to have cell phone conversations without having to physically hold a cell phone to the ear; the phone may be replaced with a headset, or the phone may be paired with a vehicle s built-in microphone and speaker system. However, hands-free systems can also support non-conversational secondary tasks, such as dialing or text messaging, by using speech-to-text (voice-recognition) technology. These systems operate by recognizing the user s spoken words and converting these into commands that can be used by the system to accomplish various tasks. In some cases, hands-free systems may rely on voice-recognition technology to initiate a task, such as dialing a phone number or searching for an address. In other cases, the entire task may rely on voice-recognition, such as the dictation of a text message to be sent to a friend. In most cases, the manual manipulation of a keyboard or input device is minimized (some systems require the push of a button or a touchscreen press to start the system), and the user s eyes are available, in theory, to 1

9 focus on driving. Additionally, hands-free systems are also more likely to comply with legislation targeting handheld cell phones. Some hands-free systems that use voice-recognition are factory installed in newer model vehicles. For example, Ford offers a hands-free system called Sync that allows not only handsfree entertainment and climate control, but also hands-free mobile phone use. To do this, Sync is paired with the driver s mobile phone and interactions with the phone are completed by interacting with Sync s hands-free interface. To initiate a call, the driver is required to state call followed by the contact s name and phone type (i.e., home or mobile), and to read text messages, the driver is required to state messages followed by listen to text messages (Ford Motor Company, 2015). Similar systems are available in many newer makes and models of vehicles. For example, Buick, Chevrolet, Chrysler, Hyundai, Mazda, Nissan, Toyota, Volkswagen, Mercedes and Volvo also offer integrated in-vehicle communication and entertainment systems that support indirect, hands-free interactions with a mobile phone (Mehler, Reimer, McAnulty, et al., 2015; Reimer, Mehler, Reagan, Kidd, & Dobres, 2016; Strayer et al., 2015B). In addition, many modern mobile phones are also equipped with personal assistant software capable of voice-recognition, opening up the potential for these devices to function as nomadic in-vehicle communication and entertainment systems in place of more expensive integrated systems. One example is Apple s Siri, a personal assistant application, which is part of Apple s mobile phone operating system (ios). Siri is capable of completing a vast variety of tasks hands-free. These include tasks that could be formerly accomplished on a cell phone with button or touch-screen presses, including sending and reading text messages, finding and reading s, creating events and reminders, setting alarms and timers, initiating calls with contacts 2

10 and answering a variety of queries (Klein, 2015). For example, a hands-free text messaging interaction may occur as follows: Driver: Hey, Siri. Text [name]. Siri: What do you want to say to [name]? Driver. How was your day? Siri. I updated your message. It says, How was your day? Ready to send it? Driver: Send. In addition to phone tasks, Siri is capable of providing navigation, completing web searches, playing selected music, setting reminders, taking notes, performing calculations, identifying songs by sound, and even reporting nearby cinema show times and making reservations at restaurants (Klein, 2015). Google and Microsoft also offer similar personal assistant applications. Google s is called Google Now and is available through the mobile Google Search application for both Android and ios powered smartphones. Microsoft s is called Cortana and is available on Windows, Android and ios phones. The development and release of new hands-free convenience technologies, and the sheer variety of ways they can be used behind the wheel, poses a problem. The dangers of using a handheld cellphone are well demonstrated within distracted driving research, and safe driving legislation banning cell phone use while driving is well warranted. Intuitively, the benefit of hands-free nomadic and integrated cell phones or other in-vehicle convenience technologies is that the driver s eyes are not required to leave the roadway, but it is not clear whether this actually translates into safety benefits. The impact of hands-free calling on driving performance has also been studied extensively (Caird, Willness, Steel, & Scialfa, 2008), but the impact of voice-recognition interactions has not. 3

11 The impact of voice-recognition interactions on driving performance can be understood from the perspective of Multiple Resource Theory (Wickens, 2002) or MRT, which is one of the prevailing theories concerning the underlying mechanisms of driver distraction. According to MRT, distraction is the result of dual-task interference, which occurs to some extent whenever two tasks are engaged in simultaneously. In order to complete a task, cognitive resources are allocated to a task. Certain resources (for example, auditory and visual resources) are assumed to be separate from one another. When engaging in two simultaneous tasks, tasks drawing from separate resources usually lead to less dual-task interference than two tasks that compete for a common resource such as vision. The concept of separate verbal and visual resources can be applied to driving while engaging in a secondary task such as voice-recognition. Moreover, driving is a task that primarily draws on visual, spatial and manual resources so any other tasks which involve (and thus compete) for the same visual, spatial or manual inputs or responses are theorized to cause interference with the driving task (Wickens, 2002). For example, the visualmanual task of texting interferes with the visual, manual and spatial task of driving. In this context, interference is indicated by a number of measures: reaction time to important events increases, lane keeping ability is compromised, and drivers do not adequately maintain their speed (Caird, Johnston, Willness, Asbridge, & Steel, 2014). The obvious benefit of hands-free nomadic and integrated cell phones or other in-vehicle convenience technologies is that the driver s eyes are not required to leave the roadway to complete secondary tasks, leaving visual resources allocated to the driving task. Based on this rationale, voice-recognition systems should interfere less with driving than visual-manual systems, such as handheld phones. However, previous meta-analyses have found that hands-free phone conversation has a significant negative impact on driving performance, despite the 4

12 theoretical separation of resources allocated to each task (Caird et al., 2008). Non-conversational tasks that rely on voice-recognition, however, have not been meta-analyzed. At this time, there is no general consensus on whether the level of interference between voice-recognition system interaction and driving has an appreciable impact on driving. The general public, however, appears to have a positive perception of hands-free technology. In fact, hands-free technologies are often viewed as safer than traditional handheld cell phones (Hamilton et al., 2013; Nurullah, Thomas, & Vakilian, 2013). Given their widespread use among drivers, determining whether hands-free systems that use voicerecognition affect driver safety is a timely issue. The purpose of this meta-analysis is to provide a clear and comprehensive understanding of how voice-recognition systems affect driving performance compared to both baseline driving and visual-manual systems. Voice-recognition systems are hypothesized to have minimal impact on driving performance compared to baseline driving. They are also hypothesized to provide driving performance benefits in comparison to visual-manual systems. An exhaustive search was conducted with the intent of capturing all experimental studies involving secondary tasks with voice-recognition systems where driving performance was measured, and results are reported in accordance with PRISMA guidelines for the reporting of meta-analyses (Moher, Liberati, Tetzlaff & Altman, 2009; see Appendix D). 5

13 Method Eligibility Criteria Participants. Participants of all age groups, nationalities and gender were sought for inclusion in the meta-analysis. Except for one study (Reimer, Mehler, D Ambrosio, & Fried, 2010), none of the included studies reported selecting participants with clinical conditions. The Reimer et al. (2010) study involved two groups a group of young drivers with ADHD, and a group of controls. Only the group of controls was included in this meta-analysis. Participant characteristics such as age, sex and nationality were coded. Descriptions of participants represented in the meta-analysis appear in Table 1. Independent variables. Tasks involving interactions with systems employing voicerecognition where the driver states a command (or series of commands) and the system interprets and acts on these commands were required in order for the study to meet inclusion criteria. General examples of voice-recognition interactions of interest include dialing or initiating a call, texting or ing, or destination entry with a navigation system. Two conditions had to be met in order to satisfy this criterion. First, the interaction with the voicerecognition system, which could be real or simulated, needed to be structured so that participants provided verbal input that the system s actions were either contingent upon, or were at least assumed to be contingent upon. Second, the voice-recognition task was required to be representative of texting, ing, navigation entry, or some other interaction type that might reasonably be engaged in with a voice-recognition system during normal driving in everyday life. Dependent variables. In order to be included in the meta-analysis, a measure of driving performance was required. Driving performance can be measured in many different ways but are 6

14 generally categorized into response time measures, discrete outcome measures (such as lane and roadway departures), and continuous outcome measures with means and standard deviations (such as steering wheel angle, lateral position and distance gap) (SAE International, 2015). Specifically, studies that measured reaction time, speed, distance gap (headway) and lateral position were targeted. These four measures are commonly reported in meta-analyses of driving performance but were chosen primarily in order to allow for comparisons specifically with Caird et al. (2008) and Caird et al. (2014), which meta-analyzed the effects of conversation and texting, respectively, on each of these dependent measures. Detection. Detection refers to whether an experimental stimulus is perceived by a participant, indicated by a specific response. It is usually measured as either the number or proportion of experimental stimuli that a participant provides responses to (i.e. hits), or the number or proportion of experimental stimuli that a participant does not provide responses to (i.e. misses). An example in a simulator setting might involve a peripheral target, such as a flashing light, which the participant has been instructed to respond to by pushing a button. Reaction time. Reaction time (RT) is defined as the time that it takes for a participant to respond to an experimental stimulus that has been detected. It is measured as a time interval, usually in seconds or milliseconds, from the start of an initiating event or stimuli to the first observable response to that event (SAE International, 2015). For example, in a simulator, the time interval between the activation of a flashing light (i.e. the onset of the event) and the participant s first observable response to the event is the participant s reaction time. It may also be measured as perception response time, which is the time required to remove the foot from the accelerator and press the brake, or it may be measured as brake response time, which is the time interval between the appearance of a hazard and a brake press (Caird & Horrey, 2011). 7

15 Speed. Travel speed is another common dependent variable, and it is often reported as mean speed, 85th percentile speed, maximum speed or variability (standard deviation) of speed (Caird & Horrey, 2011; Young, Lee, & Regan, 2009). Inability to maintain a constant velocity may be indicative of driver distraction. Driver distraction may be reflected in compensatory behaviours, such as reduced speed while engaging in a secondary task. Headway. Headway, a measure of longitudinal control, refers to the distance between a participant s vehicle and a lead vehicle during following. This distance may be measured as the gap between one vehicle s leading surface and another vehicle s trailing surface (usually the participant s vehicle), or it may refer to the gap between both vehicles rear bumpers, front axles or front bumpers (SAE International, 2015). It is usually reported as a mean headway (distance or time), minimum headway, or standard deviation of headway (Young et al., 2009). Headway is one measure used as an inference of the safety margin, or space around a vehicle, that the participant is willing to accept (Caird & Horrey, 2011; Young et al., 2009). Some research suggests that drivers may compensate for distracted driving by increasing their headway (Caird et al., 2014). Lateral position. Lateral position, also known as lane keeping, is a form of lateral control. It is often reported as mean lane position, standard deviation of lane position, or number of lane exceedances (Young et al., 2009). Lateral position refers to the position of the participant s vehicle in relation to the center of the lane (or some other lane reference) that they are traveling in (Caird & Horrey, 2011; Young et al., 2009). Standard deviation of lane position (SDLP) specifically indicates the degree of weaving within a lane, whereas mean lane position (also known as mean deviation of lane position, abbreviated MDLP) indicates the degree of 8

16 offset from a lane reference (SAE International, 2015). Inability to maintain a constant lane position may be indicative of driver distraction. Study design. Driver distraction research varies in design, but experimental, police report-based and naturalistic designs predominate. In order to satisfy inclusion criteria, studies were required to be experimental in nature. As in any experimental design, both an experimental and a comparison condition were required. For each experimental condition (i.e. the voicerecognition or VR condition), participants must engage in a secondary task with a voicerecognition system while simultaneously engaging in a driving task. However, this meta-analysis is unique in that two types of comparison conditions were targeted: visual-manual (VM) comparison, and baseline (BL) driving. In a visual-manual comparison condition, participants engage in an analogous secondary task with a visual-manual system (i.e., a traditional handheld phone or integrated system utilizing touchscreens, buttons or knobs) instead of a voicerecognition system. Including this condition allows inferences to be made about whether voicerecognition systems have any driving performance benefit over traditional handheld phones or integrated in-vehicle systems that lack voice-recognition capabilities. A baseline driving condition has participants complete comparison driving roadway segments without engaging in the secondary task. Including this condition allows inferences to be made about the impact of voice-recognition systems relative to just driving. Exclusion Criteria Tasks and/or studies where drivers merely provided vocal responses that were not assumed to be useable system inputs were excluded. For example, one study that failed to meet inclusion criteria required participants to listen to navigation, or news messages relayed via a hands-free interface while driving (Lai, Cheng, Green, & Tsimhoni, 2001). After listening to 9

17 each message, participants were required to answer three questions about the purpose or tone of the message and about details contained in the message. Although participants gave vocal responses, these responses were not treated as inputs in a voice-recognition system. They were not heard by the system, nor were they processed by the system, and they did not lead to any system outcomes specific to the vocal response. Tasks that were not representative of texting, ing, navigation entry, or some other interaction type that might reasonably be engaged in with a voice-recognition system during normal driving were excluded. Studies that failed to meet inclusion for this reason typically used cognitive tasks (e.g., a working memory task) and did not appear concerned with whether the task was analogous to an everyday secondary task, such as texting or ing. In addition, tasks that involved isolated conversation were excluded because meta-analytic reviews of cell phone conversation on driving performance have been published elsewhere (see Caird et al., 2008). All excluded studies, and their rationale for exclusion, are detailed in Figure 1. Information Sources A subject librarian was consulted in order to develop a targeted list of electronic databases to search for studies. Search terms were entered into PsycINFO, SPORTDiscus, Academic Search Complete, PubMed, Medline, TRID and Scopus with no limitations on publication year. A preliminary search was conducted using Google Scholar, which was searched with combinations of two to three of the aforementioned search terms. In addition, personal collections of studies were searched, reference lists of included studies were searched, and authors and experts were consulted for unidentified studies. 10

18 Search Strategy The search terms driver, driving, performance, behavior, behaviour, voice recognition, voice-recognition, speech recognition, speech-recognition, voice to text, voice-to-text, speech to text, speech-to-text, handsfree and hands-free were used. Search terms were combined with Boolean identifiers, which are described in detail in Appendix A. Study Selection All identified studies were imported into Mendeley Desktop Version , and duplicate citations were removed. All remaining abstracts were then screened against the inclusion criteria that are discussed above. Studies that passed the abstract screening process were then subjected to full-text review against the inclusion criteria. Once citation duplicates were removed, studies that reported unique, original data were identified. If a review paper reported a relevant study, the original study was searched for and obtained. When multiple references seemed to contain the same data, the most complete reference was retained. For example, when a conference paper and a journal publication contained the same data, typically the later journal publication was included and the conference paper excluded. Similarly, when an exhaustive technical report and a conference paper were both identified with the same data, the most complete or most interpretable data was included. Finally, when participant data appeared to be reanalyzed and/or re-reported in separate papers, care was taken to extract relevant data only once. In cases where it was unclear whether two papers reported on the same participant sample or the same experiment, authors were contacted to determine if the papers used the same data multiple times. 11

19 Data Collection All data collection was completed by one coder and added to an electronic coding database consisting of spreadsheets. Confidence ratings were generated for more subjective items (see Data Items, next section). Additionally, when difficulty or uncertainty was encountered during coding, the item in question was discussed with a second coder (JKC) until a consensus could be achieved. Data Items First, general publication and demographic information was collected. General publication information included the title of the study, the year of publication, the lead author, and the source (i.e. journal article, conference proceeding, etc.). Demographic information included the sample size, the number of males and females in the sample, the age of the sample (including mean, standard deviation and range), the source of the sample (i.e. convenience sample, employees, etc.) and the nationality of the sample. Next, methodological information was collected. Brief descriptions of the baseline driving task, the voice-recognition task and comparison task were collected, as well as the type of voice-recognition system (integrated in-vehicle or mobile phone), the type of visual-manual system (integrated in-vehicle or mobile phone), the research setting (simulator, test-track or onroad), the experimental design (within- and/or between-subjects factors, including random assignment and/or counterbalancing), and whether the participant was allowed to choose when to interact with a voice-recognition system. Confidence ratings were generated for adjudications of whether random assignment or counterbalancing occurred, as well as for whether participants were allowed to choose when to interact with the system. 12

20 Finally, extensive descriptions of voice-recognition systems were collected. First, the interaction type was coded, including whether it involved music selection, climate control, navigation, ing, text messaging, or some other type of interaction that might reasonably be expected to occur with the use of a voice-recognition system. Next, input types were coded, including whether the system utilized true speech recognition or whether a Wizard-of-Oz approach was taken (i.e., a person filled in for the system), and whether a button press was required to start the system. All of these items were accompanied by confidence ratings. Finally, output types were coded, including whether the system provided auditory feedback, visual feedback, a combination of auditory and visual feedback, or no feedback. This was accompanied by a confidence rating. Standardized mean difference effect sizes were computed from means, standard deviations, F values, t values and p values extracted from included studies and stored in an electronic coding database. When possible, effect sizes were computed from reported means and standard deviations with the following formula designed for repeated measures (Lakens, 2013), which were used in all included studies: Cohen s d average = M experimentral M comparison ( SD experimental +SD comparison 2 ). (1) These d effect sizes were then converted to r using one of two formulae. When experimental and comparison condition sample sizes were equal, the following formula was used (Steel, 2016): r = d 2 (N 2) N+( d 2 )2. (2) In cases where sample sizes differed between experimental and control conditions, a slightly different conversion formula was used (Cooper, Hedges, & Valentine, 2009, p. 234), 13

21 r = d d 2 +a (3) where a is a correction factor for unequal sample sizes and n is the sample size for each of the two groups being compared (i.e. experimental and comparison), a = (n 1+n 2 ) 2 n 1 +n 2. (4) It should be noted that although all studies utilized repeated measures, not all studies utilized purely within-subjects designs. For example, age was commonly treated as a between-subjects factor. In many cases, statistical data could be extracted from results collapsed across age or other between-subjects factors. In other cases, results were reported separately for two or more independent groups, leading to the contribution of more than one effect size. However, in one notable case, effect sizes were computed using a formula designed for between-subjects designs. Harbluk et al. (2007) utilized a study design where 32 participants were assigned to either a speech-based interface condition or a visual-manual interface condition, and participants in each condition completed four experimental conditions and three baseline drives in a counterbalanced order. For this study, effect sizes for comparisons of voice-recognition interactions and baseline driving were computed using Equation 1, using data from only the 16 participants assigned to the speech-based condition. Effect sizes for comparisons of voice-recognition interactions and visual-manual interactions, however, were computed using the following formula from Lakens (2013), using data from all 32 participants across the two conditions: Cohen s d = M experimental M comparison (n experimental 1)SD 2 experimental +(n comparison 1)SD 2 comparison n experimental +n comparison 2 (5) These effect sizes were then converted to r using Equation 2. 14

22 Across all included studies, relevant data was sometimes reported in figures and not in tables. In some cases, means and standard deviations could be extracted directly. In other cases, standard deviations needed to be derived from standard error bars or confidence interval bars. Standard deviation can be computed from standard error bars using the following formula, SD = SE N. (6) For 95% confidence intervals with large sample sizes (i.e., 100 or more participants), the following formula is recommended for deriving standard deviations (Cochrane Collaboration, 2011c), SD = N (upper limit lower limit)/ (7) When sample sizes are small or moderate, however, the divisor from Equation 6 (3.92) should be replaced with a new divisor, consisting of a t value based on the 95% confidence interval and the sample size, multiplied by two (Cochrane Collaboration, 2011c). When means and standard deviations were unavailable, r was computed directly from F and t values (when available) using the following two formulae, respectively (Steel, 2016): F r =, (8) F+df error t2 r =. (9) t 2 +df In cases where means were available but standard deviations were not given, missing standard deviations were replaced using regression imputation (Cooper et al., 2009, pp ; The Cochrane Collaboration, 2011b). Standard deviations were regressed against their associated means using the Missing Value Analysis function in SPSS (v. 23). For performance measures where regression imputation was used, sensitivity analyses were conducted (Cochrane 15

23 Collaboration, 2011b). In these analyses, the magnitude of the meta-analyzed effect sizes were inspected for appreciable differences arising from the inclusion or exclusion of effect sizes computed with imputed standard deviations. Risk of Bias in Individual Studies When assessing risk of bias in individual studies, the preferred tool is the Cochrane Collaboration s risk of bias assessment tool for randomized trials (Cochrane Collaboration, 2011f). This tool is used to assess selection bias, performance bias, detection bias, attrition bias and reporting bias. However, this particular tool is designed for randomized controlled trials, and it has some limitations when applied to driving performance studies that utilize repeatedmeasures designs. For example, selection bias may arise in a randomized controlled trial when the rule for allocating interventions to participants is not random, or when upcoming allocations are not concealed from those involved in enrolment (Cochrane Collaboration, 2011f). In a purely within-subjects design, however, each study condition is completed by all participants; thus, neither sequence generation (i.e., a rule for allocating interventions, based on a random process) nor allocation concealment are necessary. Only a few of the included studies used betweensubjects factors requiring random assignment or conducted analyses on participant subsamples. Detection bias is also unlikely to appear in a typical within-subjects driving performance study. Detection bias refers to systematic differences between groups in the way that outcomes are scored (Cochrane Collaboration, 2011e). For example, knowledge of which experimental condition a participant is assigned to (i.e. using a voice-recognition system or baseline driving) may impact evaluators visual judgments of whether or not a lane encroachment has occurred. To prevent detection bias, assessors are blinded to the knowledge of which study condition the participant is assigned to (Cochrane Collaboration, 2011e). Considering a typical driving 16

24 performance study conducted in a driving simulator, detection bias is inherently unlikely; although it may be challenging to blind outcome assessors to each experimental condition (i.e., using a voice recognition system or baseline driving), the capture of performance data for performance measures are usually automated and thus not subject to human biases. Performance bias, on the other hand, is likely unavoidable in a typical within-subjects driving performance study. Performance bias refers to systematic differences between groups with regard to the way that researchers interact with participants, based on knowledge of their group allocation (Cochrane Collaboration, 2011e). Performance bias is best prevented by blinding study participants and personnel to the knowledge of which intervention or study condition they were assigned (Cochrane Collaboration, 2011e). Although it may be possible to blind researchers to the experimental condition (i.e., using a voice-recognition system, or baseline driving), it is unlikely that the same can be done to the participant. Two sources of bias identified in the Cochrane Collaboration s tool are highly relevant to within-subjects designs. Following the collection of data, analysis may be biased when decisions about including and excluding data points or participants are made based on knowledge about participant assignment. For example, decisions may be made to exclude what may be deemed extreme values or outliers, with the knowledge of which condition these values were drawn from. Or, participants may drop out of the study, for example due to simulator sickness. Either way, withdrawals need to be reported in order to avoid attrition bias, which specifically refers to systematic differences in withdrawals between groups (Cochrane Collaboration, 2011e). Furthermore, reporting of the results may be biased for example, statistically significant differences between conditions are more likely to be reported than non-significant differences. 17

25 This is known as reporting bias (Cochrane Collaboration, 2011e), and it is a mechanism by which publication bias occurs. Unfortunately, there are currently no preferred tools for use in assessing risk of bias in experimental studies with within-subjects designs. Thus, a modified risk of bias assessment tool, based on the Cochrane Collaboration s risk of bias assessment tool for randomized trials, was developed for the present meta-analysis. The potential for selection bias, attrition bias and reporting bias were assessed for all studies at the outcome level. Detection bias was not assessed. Due to inherent limitations in within-subjects designs, performance bias may be a risk in all included studies, but it is unclear what this level of risk is. Study Quality Study quality is unique from bias study quality refers to the standard at which a study is conducted, whereas bias reflects how believable the results of a study are (Cochrane Collaboration, 2011d). There is no consensus on how exactly study quality should be assessed. However, previous meta-analytic studies (i.e. Caird et al., 2008) have noted that the quality of reporting in driving performance studies is sometimes poor and frequently depends on the experimental sophistication of the researchers. For example, Caird and colleagues (2008) found that many studies considered for their meta-analysis failed to report methodological details and statistical information adequately, leading to the inclusion of only 31% of the original set. Thus, a focus was placed on assessing the quality of reporting of methods, measures and statistical analyses across all identified studies. Specifically, studies were assessed for whether the voicerecognition apparatus was described sufficiently to allow reproduction of the study, as well as for whether steps were taken to avoid practice effects (i.e. by counterbalancing), and finally for adequate reporting of statistical information sufficient to allow effect sizes to be calculated. 18

26 Data Synthesis Effect sizes in the form of r, converted from standardized mean differences (Cohen s d), were meta-analyzed using a macro-enabled Microsoft Excel spreadsheet, Meta-Excel (Steel, 2014). Sensitivity analyses were conducted concurrently using Meta-Excel. Relevant output provided by Meta-Excel includes meta-analyzed effect sizes (both r and d), confidence intervals, credibility intervals and funnel plots. Additional meta-regression analyses were conducted using IBM SPSS Statistics Version 23. Planned analyses included testing whether sample size was predictive of effect size (publication bias) and whether study setting was predictive of effect size. Risk of Bias Across Studies Publication bias was assessed across studies using funnel plots and tests of funnel plot asymmetry. Funnel plots were generated with Meta-Excel, and tests of funnel plot asymmetry were computed in SPSS using weighted least squares (WLS) regression. For WLS regression, sample size was entered into the regression model as a predictor, with effect size r as the criterion. Several performance variables had fewer than ten studies, specifically mean deviation of lane position, headway, headway variability and speed variability for voice-recognition versus baseline driving comparisons, and detection, mean deviation of lane position, headway and headway variability for voice-recognition versus visual-manual comparisons (see Table 2). Tests for asymmetry were not conducted for these variables because these tests are discouraged when there are fewer than ten studies (Cochrane Collaboration, 2011a). 19

27 Results Included Studies Of 817 identified citations, 59 studies met inclusion criteria. Of these 59 studies, 43 were ultimately included, and the remaining 16 were excluded. For 12 of these excluded studies, there were insufficient descriptions of materials, methods or statistical analyses such that no data could be extracted for the purposes of a meta-analysis. The remaining four excluded studies reported only binary outcome performance measures, specifically lane errors and speed errors. There were not enough studies to meta-analyze these performance measures. A brief overview of each included study is presented in Table 1. Table 1 Overview of Included Studies Study Setting N Age (SD) and Sex Voice-Recognition Tasks Angell et al., 2006 (Laboratory) Angell et al., 2006 (Interstate Highway) Angell et al., 2006 (Test Track) Beckers et al., 2014 Bruyas et al., 2009 Simulator 50 Age: 7 in 20's, 10 in 30's, 9 in 40's, 8 in 50's, 9 in 60's, 7 in 70's; Sex: 26 M On-road 101 Age: 17 in 20's, 17 in 30's, 16 in 40's, 19 in 50's, 18 in 60's, 14 in 70's; Sex: 49 M Test-track 64 Age: 11 in 20's, 11 in 30's, 12 in 40's, 11 in 50's, 11 in 60's, 8 in 70's; Sex: 31 M Simulator 24 Age: M = 25.0 (2.6); Sex: 12 M Simulator 30 Age: M = 34 (11), 18-50; Sex: 15 M Number dialing. Number dialing. Number dialing. Navigation input. Using a hands-free answerphone. Included Performance Measures Detection, RT, SDLP. Detection, Headway, RT, SDLP, Speed. Detection, Headway, RT, SDLP, Speed. Detection, RT, SDLP. Detection, RT. 20

28 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Carter & Graham, 2000 Cooper et al., 2014 Cuřín et al., 2011 Graham & Carter, 2001 Greenberg et al., 2003 Harbluk et al., 2013 Harbluk et al., 2007 Simulator 32 Age: M = 29.3 (8 young males), M = 66.4 (8 old males), M = 30.0 (8 young females), M = 59.4 (8 old females); Sex: 16 M On-road 36 M = 28.1 (3.89), 22-36; Sex: 18 M Simulator 28 Age: 18-55; Sex: 14 M Simulator 48 Age: M = 35.2, 20-50; Sex: 27 M Simulator 63 Age: (48 participants), (15 participants); Sex: 32 M Simulator 16 Age: M = 29.4, 21-46; Sex: 8 M Simulator 32 Age: M = 34, (16 visual-manual interface users), M = 33, (speech-based interface users); Sex: 13 M (16 visual-manual users), 13 M (speech-based interface users) 21 Music selection, climate control, phone functions. Number dialing, contact dialing, music selection. Text messaging. Number dialing. Number dialing, voic interaction. Asking intelligent personal assistant predetermined questions, and to read texts and make calendar appointments. Navigation input. Included Performance Measures RT, Root Mean Squared Error of Lane Position (analogous to SDLP). RT. MDLP, RT, SDLP. RT, Root Mean Squared Error of Lane Position (analogous to SDLP). Detection, Headway. RT. MDLP.

29 Study Setting N Age (SD) and Sex Voice-Recognition Tasks He et al., 2014 Simulator 35 Age: M = 21.6 (3.67); Sex: 11 M He et al., 2015 Simulator 25 Age: M = (2.14), 18-25; Sex: 12 M Itoh et al., 2004 Lee et al., 2001 Maciej & Vollrath, 2009 McCallum et al., 2004 McWilliams et al., 2015 Mehler et al., 2015A (Corolla) Mehler et al., 2015B (Impala) Simulator 11 Age: M = 35.1, 25-47; Sex: 9 M Simulator 24 Age: 18-24; Sex: Not Reported. Simulator 29 Age: M = 33.2 (11.9), 19-59; Sex: 16 M Simulator 24 Age: M = 22.8, 18-35; Sex: 12 M Simulator 40 Age: M = 24.6 (2.8), (20 younger); M = 61.6 (3.4), (20 older); Sex: 20 M On-Road 48 Age: M = 39.8 (17), for 24 females, M = 40.3 (16.7), for 24 males; Sex: 24 M On-Road 48 Age: M = 41.8 (16.6), for 24 females, M = 39.6 (16.4), for 24 males; Sex: 24 M Text messaging (of phone numbers). Text messaging. Music selection, navigation input. ing. Contact dialing, music selection, navigation input. ing, internet activities. Navigation input. Contact dialing, navigation input. Contact calling, navigation input. Included Performance Measures Headway, Mean Lane Position, RT, SD Headway, SDLP, SD Speed, Speed. Detection, Headway, Mean Lane Position, RT, SD Headway, SDLP, SD Speed, Speed. SDLP. RT. MDLP, RT, SDLP. RT. SDLP, Speed. SD Speed, Speed. SD Speed, Speed. 22

30 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Mehler et al., 2014 Mehler et al., 2015C (CLA) Munger et al., 2014 Neurauter et al., 2012 On-Road 64 Age: M = (1.1), for 8 females, M = (1.3), for 8 males, M = (5.9), for 8 females, M = (5.3), for 8 males, M = (3.0), for 8 females, M = (3.7), for 8 males, M = (1.8), for 8 females, M = (4.2), for 8 males; Sex: 32 M On-Road 48 Age: M = 38.9 (15.6), for 24 females, M = 39.8 (15.3), for 24 males; Sex: 24 M Simulator 24 Age: (12 younger), 55 and over (12 older); Sex: 12 M Test-Track 24 Age: (younger), (older); Sex: 12 M Music selection, navigation input. Contact calling, navigation input. Navigation input. Navigation input, text messaging. Included Performance Measures Speed, SD Speed. SD Speed, Speed. Detection, RT, SDLP. Speed Variance. 23

31 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Ranney et al., 2005 Reimer et al., 2011 Reimer et al., 2010 Reimer et al., 2013 Test-track 21 Age: M = 40.3 (13.9), 22-67; Sex: 10 M Simulator 37 Age: M = 20.7 (0.9), (18 younger), M = 56.3 (4.5), (19 late middle age); Sex: 19 M Simulator 35 Age: M = (1.89); Sex: 20 M On-Road 60 Age: M = (3.0), for 15 younger females, M = (2.7), for 15 younger males, M = (3.0), for 15 older females, M = (2.9), for 15 older males; Sex: 30 M Baseline tasks (dialing phone numbers, tuning radio stations), Simple tasks (opening message containing list of items, creating voice memo) and complex tasks (opening message, opening phone book, autodialing automated system, retrieving information, creating voice memo). Number dialing, interacting with automated phone trees and voic systems. Number dialing, interacting with automated phone trees and voic systems. Contact dialing, music selection, navigation input. Included Performance Measures Detection, RT, SDLP. Speed, SD Speed. Speed. SD Speed, Speed. 24

32 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Reimer et al., 2015 (ACC) Reimer et al., 2016 On-Road 24 Age: M = 63.2 (2.6), for 6 older females, M = 64.8 (2.8), for 6 older males, M = 24.2 (3.2), for 6 younger females, M = 24.2 (1.8), for 6 younger males; Sex: 12 M On-Road 80 Age: M = 40.4 for females, M = 40.3 for males; Sex: 40 M Salvucci, 2001 Simulator 11 Age: M = 25, 19-32; Sex: 6 M Salvucci & Macuga, 2002 Schreiner et al., 2004 Simulator 7 Age: 18-40; Sex: Not Reported Test-track 37 Age: M = 23.4 (18 younger), M = 56.6 (19 older); Sex: 9 M (younger) 9 M (older) Contact dialing. Contact dialing. Number dialing, contact dialing. Contact dialing. Number dialing. Included Performance Measures SD Speed, Speed. SD Speed, Speed. Root Mean Squared Error of Lane Position (analogous to SDLP). Root Mean Squared Error of Lane Position (analogous to SDLP), Root Mean Squared Error of Speed (analogous to SD Speed). Detection, RT. 25

33 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Schreiner, 2006 Serafin et al., 1993 Strayer et al., 2015A (Smartphone Experiment 1) Strayer et al., 2015A (Smartphone Experiment 2) Strayer et al., 2013 (Experiment 2) Strayer et al., 2013 (Experiment 3) Strayer et al., 2015B Strayer et al., 2014 (Experiment 2) Strayer et al., 2014 (Experiment 3) Simulator 12 Age: M = 36.9, 22-53; Sex: 4 M Simulator 12 Age: M = 24, (younger), M = 70, (older); Sex: 6 M On-road 31 Age: M = 42, 21-68; Sex: 16 M On-road 34 Age: M = 42.5, 22-68; Sex: 19 M Simulator 32 Age: M = 23.5, 19-36; Sex: 22 M On-road 32 Age: M = 23.5, 18-33; Sex: 12 M On-road 257 Age: M = 44, 21-70; Sex: 127 M Simulator 41 Age: M = 25.2, 18-40; Sex: 21 M On-road 40 Age: M = 26.1, Sex: 23 M Number dialing, contact dialing. Number dialing. Number dialing, contact dialing, music selection. Text messaging. Text messaging, ing. Text messaging, ing. Number dialing, contact dialing, music selection. Climate control, ing, text messaging, navigation entry. Climate control, ing, text messaging, navigation entry. Included Performance Measures Lane Position Variance, Speed Variance. SDLP. RT, Speed. RT, Speed. Headway, RT. RT. RT. Headway, RT. RT. 26

34 Study Setting N Age (SD) and Sex Voice-Recognition Tasks Terken et al., 2011 Törnros et al., 2005 Truschin et al., 2014 Tsimhoni et al., 2004 Simulator 41 Age: 20-29; Sex: Not Reported Simulator 23 Age: Unclear (M is approximately 34, range is approximately 24 to 54); Sex: Unclear (11 or 12 M) Simulator 98 Age: Unclear (M is approximately 23, SD is approximately 5); Sex: Unclear (between 74 and 88 M) Simulator 24 Age: M = 24, (12 younger), M = 69, (12 older); Sex: 12 M Yager, 2013 Test-track 43 Age: 2 aged 16-17, 16 aged 18-24, 4 aged 25-29, 3 aged 30-39, 10 aged 40-49, 7 aged 50-59, 1 aged 60 and over; Sex: 20 M ing, text messaging. Number dialing. ing. Navigation input. Text messaging. Included Performance Measures Headway, Speed, SD Headway, SD Speed. Detection, RT, SDLP, Speed. MDLP. SDLP, SD Headway, Speed. Detection, RT, Speed. The extracted data represents 2,000 drivers of all age groups. The sample is about 52% male and primarily North American (15% of the sample is drawn from Europe, and less than one percent is drawn from elsewhere). The process of identifying, screening, including and excluding studies is illustrated in detail in Figure 1. 27

35 All identified citations (N = 817) Lab library: 34 Google Scholar: 6 Ancestry: 81 Databases: 681 Spontaneous hits during web searches: 4 Expert consultation: 11 Duplicates removed (N = 432) All unique citations (N = 385) Citations excluded during abstract review (N = 136) Abstracts unavailable (N = 1) Studies subject to full-text review (N = 248) Unable to extract data (N = 12) Studies failing to meet inclusion criteria (N = 184) Not enough data to meta-analyze (N = 4) No experimental driving task: 24 No relevant voice-recognition condition: 107 No relevant driving performance measurement: 8 Unoriginal data: 45 Full-text unavailable (N = 2) Not in English (N = 3) Figure 1. Study selection. Included studies (N = 43) 28