資料分析往年language based multimodal displays for the handover of control in autonomous carsVIP

1、Language-Based Multimodal Displays for the Handover of Control in Autonomous CarsIoannis Politis1, Stephen Brewster2 Glasgow Interactive Systems Group School of Computing Science University of GlasgowFrank Pollick School of Psychology University of Glasgow Glasgow, G12 8QB, UKFrank.Pollickglasgow.ac

2、.ukGlasgow, G12 8, UK1I.Politis.1research.gla.ac.uk 2Stephen.Brewsterglasgow.ac.ukABSTRACTThis paper presents a first evaluation of multimodal lan- guage-based warnings for handovers of control in autono- mous cars. A set of possible handover situations varying in urgency is described. A set of mult

3、imodal, language-based warnings for these situations is then introduced. All combi- nations of audio, tactile and visual warnings for handovers were evaluated in terms of perceived urgency, annoyance and alerting effectiveness. Results showed clear recognition of the warning urgency in this new cont

4、ext, as well as low perceived annoyance overall, and higher perceived effec- tiveness for critical warnings. The time of transition from self-driving to manual mode in the presence of the warnings was then evaluated. Results showed quicker transitions for highly urgent warnings and poor driving perf

5、ormance for unimodal visual warnings. These results provide a novel set of guidelines for an effective transition of control between car and driver in an autonomous vehicle.Author KeywordsMultimodal feedback; autonomous cars; handover; warn- ings; audio; visual; tactile; speech; Tactons; urgency.ACM

6、 Classification KeywordsH.5.2 Information interfaces and presentation: User Inter- faces. - Auditory (non-speech) feedback; Haptic I/O; Voice I/O.INTRODUCTIONAutonomous driving, the ability of a car to partially or fully drive itself, is gaining attention both in industry and with the public. The mo

7、ve towards autonomous vehicles is pre- dicted to have a large effect on the car market 9. This is, however, not without concerns over safety, highlighting thePermission to make digital or hard copies of all or part of this work for al or classroom use is granted without fee provided that copies aren

8、ot made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for com- ponents of this work owned by others than ACM must be honored. Ab- stracting with credit is permitted. To copy otherwise, or republish, to post on s

9、ervers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from P.AutomotiveUI 15, September 01 - 03, 2015, Nottingham, UK 2015 ACM. ISBN 978-1-4503-3736-6/15/09$15.00importance of carefully designing interfaces for autono- mous cars. Au

10、thorities like NHTSA 12 and SAE 18 cat- egorise the levels of autonomy in a vehicle. They stress that autonomy is not a binary state (no autonomy to full auton- omy) but rather a staged process that will happen over time. The next generation of vehicles will be at Level 3 Limited Self-Driving Automa

11、tion 12. There, “the driver is not constantly expected to monitor the roadway, but to still be available for occasional control with sufficiently comforta- ble transition times”. This point of transition of control is referred to as a handover. It is essential to warn the driver effectively in such

12、a handover situation and design warn- ings that will enable safe transitions. There have been some studies on handover warnings using a limited number of modalities 13, looking into message content 8 and inves- tigating transition times 5. However, there has been no study on designing effective mult

13、imodal warnings for di- verse handover situations when the driver of an autonomous car is busy with a demanding side task. The lack of work on this important topic motivated our study.Figure 1: The setup of the experiment on autonomous hando- vers. In a critical event which the car could not address

14、, a handover to the driver was enforced (a). In non-critical events, a handover to the driver was requested (b),(c). When in au- tonomous mode (d), drivers were playing a tablet game (e).DOI:.See alsofor aof the ex-perimental setup.3In this paper, we first provide a set of situations varying in urge

15、ncy that cover a range of possible transitions of control from the driver to the car and vice versa. We then present a study investigating a set of warnings designed for these situations, using all combinations of audio, tactile and visu- al modalities varying in urgency. We evaluate the warnings in

16、 a driving simulator with participants playing a game on a tablet, to simulate future driving situations. We finally pre- sent a first set of guidelines to inform drivers about auton- omous car handovers in an effective manner.Multimodal Displays and UrgencyThe use of multimodal displays to alert dr

17、ivers about differ- ent road situations has shown benefits in the past 7. Since the cues should convey the appropriate level of urgency to the drivers, Baldwin & Moore 1 showed that the use of the word “Danger” increased ratings of perceived urgency for speech. “Warning” and “Caution” created interm

18、ediate results and “Notice” led to the lowest urgency ratings. Baldwin 2 also observed lower reaction times to urgent words, presented with higher signal intensity. Edworthy et al. 3 found that urgently spoken signal words were per- ceived as more urgent and appropriate.Using all combinations of aud

19、io, visual and tactile modali- ties, Politis, Brewster & Pollick 15 evaluated a set of mul- timodal abstract warnings across three levels of urgency. They found that an increased number of modalities in- creased ratings of urgency and annoyance. They then eval- uated the warnings in the presence of

20、a critical driving event and found lower reaction times 14. The use ofMerat . 11 found that drivers were quicker to resume control from highly automated driving when automation was switched off regularly compared to when automation switch-off was triggered when drivers disengaged their at- tention f

21、rom the road. In their study, the current driving mode was indicated through a text display lighting up when in automated driving. However, there was no emphasis on how to inform the driver about the transition from manual to automated modes. We address this gap in our study. Gold et al. 5 investiga

22、ted the behaviour of drivers when re- quested to return to driving due to automation failures. Drivers engaged in a tablet side task were warned through a pure tone and a visual icon that they needed to take over control due to an unexpected event and the required times for such transitions were inv

23、estigated. There was no com- parison of warning modalities in this study. It was found that when warnings were presented closer in time to an im- minent handover (5 sec before) reactions were quicker but driving behaviour was less accurate compared to when they were presented earlier (7 sec before).

24、 In our study, we used a similar distraction task on a tablet and compared a wide set of warning modalities. We also addressed scenarios where drivers resumed control and vice versa.Regarding warning design for transitions between manual and autonomous modes, literature is limited. Koo . 8 used spee

25、ch warnings when the car took over from the driv- er due to an unexpected event. They found that messages describing why an event happened (e.g. “Obstacle ahead”) were preferred and led to better driving performance than messages describing how it was addressed (e.g. “The car is braking”). Urgency i

26、n the messages was not varied and all the driving situations tested were critical. Naujoks, Mai & Neukum 13 studied the opposite case, where a driver needed to take over from a vehicle due to an automation failure. They presented audio (a pure tone) together with visual warnings (flashing icon on th

27、e dashboard) during takeover requests. They found that bimodal presentation of warnings led to shorter handover times and better driving behaviour compared to only visual presentation.As evident from the above studies, there is distinct lack of research on how to design effective warnings for contro

28、l handovers, particularly when the driver is involved in a secondary task. In our study, we address this by designing a set of language-based messages across all combinations of audio, tactile and visual modalities, taking into account the urgency of the takeover situation. We design multimodal warn

29、ings that address situations where control is handed from car to driver, as well as from driver to car. We first evaluate these warnings to assess their perceived urgency, annoyance and alerting effectiveness. We then study in a simulator the time taken and driving behaviour when re- turning to driv

30、ing from a tablet-based game, for all modali- ties and urgency levels, to gain a deeper understanding of effective warning design for autonomous cars.Speech Tactons, tactile cues mled on the structure ofspeech, was found to improve the recognition of warningsurgency and perceived effectiveness 16. F

31、inally, compar- ing a set of language-based messages to abstract ones, they found similar performance as well as a slight improvement of driving metrics when using language-based cues 17.In this paper, we build on these warnings and put them in the context of autonomous cars. No study has yet tested

32、 them in this context, yet there is need for effective warnings in situations where the car and driver will share control. Since we wish to describe why a handover occurs, we use speech to accurately inform the drivers about the events that lead to this handover.Control Handovers in autonomous carsD

33、riving under high automation is clearly different to manu-al driving. De Winter. 21 state that more non-drivingtasks are performed in highly-automated driving situations,such as using in-vehicle entertainment systems. A differen- tial effect on situational awareness can also be observed; if drivers

34、are instructed to detect objects in the environment, situational awareness increases with automated driving. This effect is reversed, however, if they are engaged in non- driving tasks. This highlights the design challenge to keep the driver attentive when automation is not complete and handovers ma

35、y occur. Interestingly, this is still a relevant topic also in the field of aviation 4.4WARNING DESIGNWe designed a set of six speech messages covering a range of possible handovers of control between the car and the driver. We used three different Levels of Designed Urgency (LDU) for the envisioned

36、 situations (Level High - LH, Level Medium - LM and Level Low - LL), as in previous studies 16,17. We also used situations where either the car would hand over control to the driver (CD) or the driver to the car (DC). This resulted to six speech messages, presented in Table 1 (LHCD, LMCD, LLCD, LHDC

37、, LMDC and LLDC).The messages used were adjusted from 10, containing a set of in-vehicle messages prioritized according to SAE J2395 19. Adjustments were in order to avoid resem- blance between messages in terms of rhythm. High priority messages in 10 were mapped to LH, intermediate priority ones to

38、 LM and low priority ones to LL. The word “Dan- ger!” was added before each LH message, “Warning!” be- fore LM and “Notice!” before LL, since this has shown to provide distinctively different urgency ratings in previous studies 1,6,16. Finally, in LH the handover was enforced, since imminent actions

39、 would be needed in such critical situations, while in LM and LL, the handover was only re- quested. This was reflected in the text of the messages, con- cluding with whether the messages were an enforced hand- over or handover request. In line with 8, the messages explained why a handover was neces

40、sary, rather than how the handover would happen.The messages were recorded by a female voice actor using a Rode NT2-A1 condenser microphone. In line with 6,16, the actor was instructed to speak messages of LH urgently, as if a loved one was in imminent danger. LM messages were spoken non-urgently, a

41、s if in a friendly conversation with nothing interesting about the situation and LL messag- es were spoken in a monotone, deadpan manner. LH mes- sages were slightly modified to remove pauses between sentences so as to decrease duration. As tactile equivalents of the audio warnings, we used Speech T

42、actons with a C2 tactor2, which were constructed following the procedure described in 16. Finally, for the visual warnings, the text of the warnings was displayed for the duration of the utter- ance and varied in colour, in line with 17 (Red for LH, Orange for LM and Yellow for LL3). For all modific

43、ations, Praat4 and Audacity5 software were used. Although there are possible limitations of using speech messages as alerts (e.g. see 17), we saw benefit in creating a good initial set of warnings, which were not available in this use case.We presented the designed warnings in all combinations of th

44、e audio, visual and tactile modalities: Audio (A), Visual (V), Tactile (T), Audio + Visual (AV), Audio + Tactile (AT), Tactile + Visual (TV), Audio + Tactile + Visual (ATV). As a result 42 different cues were created6, 7 cues with all modalities (A, T, V, AT, AV, TV, ATV) 3 Levelsof Designed Urgency

45、 (L , L , L ) 2 Situations (CD, DC).HMLThese warnings were evaluated in two experiments, looking into subjective and objective responses of participants when exposed to the cues.EXPERIMENT 1: RATING THE DESIGNED WARNINGSThe first experiment investigated subjective responses of participants when expo

46、sed to the warnings. This has never been studied before and is essential in order to provide in- sights on how such warnings would be perceived. A 732 within subjects design was used with Modality, LDH and Situation as the independent variables and Perceived Ur- gency (PU), Perceived Annoyance (PA)

47、and Perceived Alerting Effectiveness (PAE) as the dependent ones.ProcedureTwenty one participants (3 female) aged between 18 and 29 years (M = 21.00, SD = 2.84) took part in this experiment. They all held a valid driving license and had between 1 and 8 years of driving experience (M = 3.36, SD = 2.0

48、1). All were right handed University students and reported normal vision and hearing. The experiment took place in a Univer- sity room where participants sat in front of 27-inch Dell 2709W monitor and a PC running the experimental driving Handover of Control12Table 1: The messages designed, using si

49、tuations of High (L ),HhtmlMedium (LM) and Low urgency (LL). The handovers of control were from Car to Driver (CD) or from Driver to Car (DC). For each message the duration (D), peak (P) and average fre- quency (AF) of the audio are reported.3 Red was RGB(255,0,0), Orange was RGB(255,127,0) and Yell

50、ow was RGB(255,255,0).45hum.uva.nl/praat/net/6 All warnings can be found in5UrgencyCDDCLHDanger!Danger!Collision ImminentObject in roadway You have control!I have control!D: 2.7 secD: 3.0 secP: -0.0 dBFSP: -0.2 dBFSAF: 371 HzAF: 346 HzLMWarning!Warning!GPS signal weakDense fog aheadWant to take over

51、?May I take over?D: 3.8 secD: 3.4 secP: -11.0 dBFSP: -9.4 dBFSAF: 309 HzAF: 296 HzLLNotice!Notice!Toll ahead, 5 poundsNewfrom John Want to take over?May I take over?D: 4.7 secD: 4.2 secP: -18.5 dBFSP: -20.4 dBFSAF: 212 HzAF: 211 Hzsimulator. Auditory cues were displayed through a set of Sennheiser H

52、D 25-1 headphones. Tactile cues through a wristband on participants left hand with a C2 tactor at- tached to it, in line with 16. Visual cues were coloured text appearing for the duration of the utterance of the audio in the top centre of the screen, simulating a Head up Dis- play (HuD), in line wit

53、h 17. They were sized 228700 pixels (about 721 cm). Participants provided all responses using a mouse. To cover any tactor noise, car sound was played during the experiment.After being welcomed and explained the experimental pro- cedure, the 42 cues were displayed in a random order to participants t

54、o familiarize them with the signals. For each cue, they could either repeat it or proceed to the next when they felt familiar with it. Afterwards, they were again pre- sented with the cues when sitting in front of a driving simu- lator software depicting a rural road scene with a straight road and a

55、 car in front. This software has been used in many studies, e.g. 14. The participants car was self- driving. They were asked to imagine they were sitting in the drivers seat of an autonomous vehicle, wearing a wrist mounted device like a smart watch for vibration. Partici- pants rated all cues in te

56、rms of PU, PA and PAE, by com- pleting a 5-point Likert scale, in line with 2,16. In all rat- ings, the scale was labelled: Not at all (1), Slightly (2), Moderately (3), Very (4) and Extremely (5). Each cue was presented twice, resulting to 82 trials.ResultsPerceived UrgencyData for PU were analysed

57、 using a three-way repeated measures ANOVA, with Modality, LDU and Situation as factors. Due to sphericity violations, degrees of freedom were corrected using GreenhouseGeisser estimates. There was a significant main effect of Modality (F(3.20,131.25) = 29.88, p 0.001). Contrasts revealed that modal

58、ities were rated for PU in the following order7: T and V lower than A (F(1,41) = 20.11, r = 0.57, p 0.001), A lower than TV and AV (F(1,41) = 7.80, r = 0.40, p 0.05), TV and AV lowerthan AT (F(1,41) = 5.51, r = 0.34, p 0.05) and AT lower than ATV (F(1,41) = 7.62, r = 0.40, p 0.05). See Figureabc2.a for mean values ofcross modalities. There was asignificant main effect of LDU (F(1.58,64.56) = 306.02, p 0.001). Contrasts revealed that levels were rated in the following order: LL lower