D S R G. Alina Mashko. Measuring tools and hardware. Department of vehicle technology. Faculty of Transportation Sciences

Transcription

1 Measuring tools and hardware Alina Mashko Department of vehicle technology Faculty of Transportation Sciences Czech Technical University in Prague

2 Measurement on driving simulator Main benefits: Safe, close to realistic environment Possibility to tailor measurement per given task Assessment of wide range of simulated situations Testing of scientific hypotheses

3 Full simulator based on Škoda Superb The first big involving full car body Based on WV simulation systems Steady based simulation system Full car 360 DEG FOV 270 projection system hexagonal without blending plus LCD mirrors Target research drowsiness

4 Measurement setup basic data inputs

5 Measured quantities technical and psycho-physiological data are collected all data are synchronized so that it is possible to do further correlation analysis on them not all of them are suitable for the analyses Objective measurement Technical Speed Trajectory Lane change Movements of foot pedals Movements of steering wheel Proband Subjective measurement Human Related EEG ECG EOG Reaction time Movements of head Camera record Questionnaire NASA TLX Expert s appraisal

6 Electroencephalography (EEG) Electroencephalography (EEG) is the recording of electrical activity along the scalp produced by the firing of neurons within the brain obtained by placing electrodes on the scalp with a conductive gel or paste electrode locations and names are specified by the International system amplify the voltage between the active electrode and the reference (typically 1, ,000 times - human EEG signal is about 10µV to 100 µv) signal is digitized via an analog-to-digital converter (sampling typically occurs at Hz)

7 Electroencephalography (EEG) The EEG is typically described in terms of rhythmic activity (is divided into bands by frequency) transients. frequency bands are usually extracted using spectral methods most of the cerebral signal falls in the range of 1 20 Hz activity below or above this range is likely to be artifactual one second of EEG signal

8 Electroencephalography (EEG) (wave patterns) Delta (up to 4 Hz) it tends to be the highest in amplitude and the slowest waves it is seen normally in adults in slow wave sleep it is also seen normally in babies Theta (4 Hz to 7 Hz) it is seen normally in young children it may be seen in drowsiness or arousal in older children and adults it can also be seen in meditation excess theta for age represents abnormal activity

9 Electroencephalography (EEG) (wave patterns) Alpha (8 Hz to 12 Hz) the "posterior basic rhythm" (also called the "posterior dominant rhythm" or the "posterior alpha rhythm") it is seen in the posterior regions of the head on both sides, higher in amplitude on the dominant side it emerges with closing of the eyes and with relaxation it attenuates with eye opening or mental exertion Beta (12 Hz to about 30 Hz) it is seen usually on both sides in symmetrical distribution it is closely linked to motor behavior and is generally attenuated during active movements it is often associated with active, busy or anxious thinking and active concentration (low amplitude with varying frequencies ) rhythmic beta is associated with various pathologies and drug effects

10 Electroencephalography (EEG) (wave patterns) Gamma (approximately Hz) gamma rhythms are thought to represent binding of different populations of neurons together into a network for the purpose of carrying out a certain cognitive or motor function gamma waves may be implicated in creating the unity of conscious perception (the binding problem) Mu (SMR) (8-13 Hz) and partly overlaps with other frequencies it reflects the synchronous firing of motor neurons in rest state the meaning of SMR is not fully understood a person is producing a stronger SMR amplitude when the corresponding sensory-motor areas are idle SMR typically decrease in amplitude when the corresponding sensory or motor areas are activated neurofeedback training can be used to gain control over the SMR activity

11 Type Frequency (Hz) Delta up to 4 Theta 4 7 Hz Alpha 8 12 Hz Comparison of EEG bands Location Normally Pathologically frontally in adults, posteriorly in children; high amplitude waves found in locations not related to task at hand posterior regions of head, both sides, higher in amplitude on dominant side. Central sites (c3- c4) at rest. both sides, symmetrical distribution, most Beta Hz evident frontally; low amplitude waves Gamma somatosensory cortex Mu 8-13 Hz sensorimotor cortex adults slow wave sleep in babies has been found during some continuous attention tasks young children drowsiness or arousal in older children and adults idling associated with inhibition of elicited responses (has been found to spike in situations where a person is actively trying to repress a response or action) relaxed/reflecting closing the eyes Also associated with inhibition control, seemingly with the purpose of timing inhibitory activity in different locations across the brain alert/working active, busy or anxious thinking, active concentration Displays during crossmodal sensory processing (perception that combines two different senses, such as sound and sight) Also is shown during short term memory matching of recognized objects, sounds, or tactile sensations Shows rest state motor neurons subcortical lesions diffuse lesions metabolic encephalopathy hydrocephalus deep midline lesions focal subcortical lesions metabolic encephalopathy deep midline disorders some instances of hydrocephalus coma benzodiazepines A decrease in gamma band activity may be associated with cognitive decline, especially when related the theta band; however, this has not been proven for use as a clinical diagnostic measurement yet Mu suppression could be indicative for motor mirror neurons working, and deficits in Mu suppression, and thus in mirror neurons, might play a role in autism.

12 Electroencephalography (EEG) During the recording, a series of activation procedures may be used: hyperventilation photic stimulation (with a strobe light), eye closure, mental activity, sleep and sleep deprivation Works of our laboratories on different ways of analysis of the EEG: classical frequency analysis, with advanced fuzzy classification nonlinear methods like LLE (Largest Lyapunov exponent), chaotic attractors classification using neural networks

13 Eye tracking Eye tracking is the process of measuring eye behavior parameters: Fixations Saccades Blinks Eye closures used in: cognitive science psychology human-computer interaction (HCI) marketing research and medical research, etc.

14 Eye tracking (iview X ) Systems we use: iview X HED iview X RED

15 Composition (iview X HED): 2x mini camera 1 st - for real-time scene (gaze) capturing 2 nd - for real-time eye picture capturing 1x IR emitor (eye pupil reflection) 1x IR receiver 1x half transparent IR mirror

16 Eye tracking (iview X HED) iview X HED C o m p i l a t i o n S W calibration raw data files real time operating video sequences savings

17 Eye tracking (iview X HED)

18 Eye tracking (iview X HED)

19 Eyetracker s outputs Video file cursor showing eye s focus overlapped over the picture from scene camera Idf file - containing coordinates of gaze point in each frame of video file from scene camera

20 Eye tracker Output examples

21 Eye-tracker Smart Eye Pro Fixed Non-contact Multiple cameras installation IR flash Camera

22 Eye behavior Eye behavior is characterized by saccadic movements interrupted by fixations. Saccades: - Amplitude is normally <15 degrees (85%) - Duration is normally ms - Duration count: ms for larger than 5 degrees saccades + 2 ms for every amplitude

23 Saccades metrics The red line indicates the position of a fixation target and the blue line the position of the fovea. When the target moves suddenly to the right, there is a delay of about 200 ms before the eye begins to move to the new target position (After Fuchs, 1967.) Forvea - area of the retina specialized for high acuity in the center of the macula; contains a high density of cones and few rods.

24 Small pursuit metrics The metrics of smooth pursuit eye movements. These traces show eye movements (blue lines) tracking a stimulus moving at three different velocities (red lines). After a quick saccade to capture the target, the eye movement attains a velocity that matches the velocity of the target. (After Fuchs, 1967.)

25 Eye behavior - fixations Take place between saccades Correspond to brain encoding image seen when eye is fixed on target Duration of fixation - ~100 ms to 1 sec (depends on task). Smooth pursuit fixation moving with a velocity of a moving target Miniature eye movement occurs during fixation: tremmor, drift and microsacades

26 Glance metrics (ISO 15007, SAE J-2396) Number of glances Total glance time Glance duration Percent time on AOI (area of interest) Glance location probability Distribution of gaze duration with three different positions of display (a-c) and baseline (d). Source: Chenjiang Xie et. al.

27 Glance metrics (ISO 15007, SAE J-2396) Derived glance measures: Mean glance duration Maximum glance duration Total eyes off road time (TEORT) Percentage eye of road time (PEORT) Percentage of transition times

28 Measures technical data not dependent on subject of the driver (the simulator is always the same) they can be easily collected with low noise ratio measures: 1. lane departure 2. lane position variability 3. variation of actual lateral deviation from a geometrical lane centerline 4. speed fluctuation 5. steering wheel movements 6. reaction time

29 Lateral position

30 Distance [m] Distance [m] Lane departure trajectory fluctuations around geometrically ideal path (the so called weaving ) blue line - distance of the center point of the virtual car and geometrically ideal center of the lane black line - lane border (in this case central line) driver loaded (or disturbed) with secondary task produces many more lane departure events Time [ms] x 10 the deviation of a car trajectory from geometrically ideal path without disturbing Time [ms] x 10 5 the deviation of a car trajectory from geometrically ideal path with disturbing

31 percent occurrence D S R G percent occurrence percent occurrence percent occurrence Lane position variability the analysis compares the parts where the driver was forces to fulfill secondary tasks with those parts where he/she was driving without any without disturbing 0.4 distraction 0.2 the histogram is significantly flatter and covers bigger range when computed for driving under disturbance without disturbing Mean value = Median value = Variance = with disturbing Mean value = Median value = Variance = with disturbing normalized histograms of path deviations without disturbing Mean value = Median value = Variance = normalized histograms of path deviations with disturbing

32 Variance (with disturbing) [m 2 ] Variance (with disturbing) [m 2 ] Variation of actual lateral deviation standard deviation (or variation) is frequently used as a main measure variability of lateral deviation increases with drop of driver s performance car trajectory variation computed as a discrete distance from actual car center position from the reference trajectory curve the lower variation -> fluent drive without much of line crossings the ratio of loaded and non-loaded parts (Y axis) tells us about significant degradation of performance of the second group 1.4 Men <60 years old 1.4 Men >60 years old Variance (without disturbing) [m 2 ] Variance (without disturbing) [m 2 ] ratios between variations when driving with vs. without secondary task (light and dark points represent different parts of the testing track)

33 Lane departure area calculation

34 Speed fluctuation the drivers were instructed to keep the speed (for example 50 km/h or 90 km/h) depending on the driving circuit part it was expected that the speed fluctuation would be more apparent when the driver is loaded the speed is corrected by the driver a just after he/she finishes the operation with a device normalized histograms of speed fluctuation with disturbing normalized histograms of speed fluctuation without disturbing

35 Distance [m] Weaving and speed typical development (drowsy drivers) 12 Distance Linear Time [ms] x 10 5 Continuous increase Steady behavior or decrease Driver cannot keep the speed in reasonable boarder / goes much faster than required 60,87% 17,39% 21,74%

36 Speed deviation

37 Speed deviation (maximized view)

38 Speed deviation tested cohort

39 Lane departure area tested cohort

40 Steering wheel movements number of correction movements done by driver on the steering wheel was counted with respect to time the data were obtained form experiments focused on a detection of fatigue, therefore the driver was not loaded from any other source percentage of the fast corrections related to all instant corrections (nonincreasing trend) percentage of the fast corrections related to all instant corrections (increasing trend)

41 Measuring time of reaction Triggered event Example: traffic light signal, pedestrian, overcoming vehicle, wild animal etc. Time of reaction is here time from seen trigger is launched (or seen - individual) to reaction (break pedal deployment, gas pedal release, full stop of vehicle) Image source: ArriveAlive.co.za

42 Triggers in driving scenario

43 Reaction time results example Average reaction time for each trigger event in each state: green - fresh, light blue - sleep deprived 1st hour, dark blue - sleep deprived 2nd hour.

44 Visual features measure Application in experiments for driver sleepiness Body and head movements Head tilt and rotations Head and shoulders shake Body stretching, readjusting sitting position Touching face (rubbing nose, eyes) Eye behavior Eye closures (blinking and longer closures) Direction of sight (control panel/road) Mimics (face) Stretching face muscles Fresh looking vs. relaxed face expression

45 Fixation of driver initial state Straight seated driver Eyes are wide (but naturally) open Driver looking on the road and dashboard with regular/usual frequency Blinking is normal (yet, often increased as compared to person after regular sleep) Both hands on wheel (upper part) Car is in the middle of the traffic lane k616.fd.cvut.cz Driving Simulation Research Group FTS-CTU

46 Initial state, k616.fd.cvut.cz Driving Simulation Research Group FTS-CTU

47 Distinguished patterns of drowsy behavior Increasing of blinking frequency, prolonged blinks eye closures sleep. Adjustment of self in a seat seeking for more comfortable position Leaning back or to the sides Relaxation of body muscles shoulders and gradually the whole body go down Relaxation of face muscles Lip biting and lip licking Forced straightening or stretching of body Hands on the wheel go down from typical initial 10/2 o clock position Vehicle control loss (weaving up to lane departure) k616.fd.cvut.cz Driving Simulation Research Group FTS-CTU

48 Example: closing eyes and lane departure k616.fd.cvut.cz Driving Simulation Research Group FTS-CTU

49 Video example

50 Eye measures for sleepiness Percentage of eye closure (PRECLOS) a measure of the percentage of eyelid closure over a period of time Blinking rate Blinking frequency and duration Gaze fixation rate Gaze fixation duration (average)

51 Eye measures - Blink count

52 Subjective measures Questionnaires User experience User rating User acceptance Individual perception Expert (observer) assessment Rating per predefined scale Assessment of behavior based on expert prior knowledge Self assessment (self-rating) Typically, according to some scale

53 Expert assessment for sleepiness ORS Observer rated sleepiness (the scale is on the next page) Several researchsers observe participant during driving sessions (online). (A. Anund et. al., 2013) Observers can also provide evaluation of video records pf driving sessions (offline).

54

55 ORS (Ahlstrom et.al., 2015) ORS 0: Alert Blink: normal Yawn: no Body position: sitting still Body movements: normal ORS 1: Sleepy, no effort to stay awake Blink: sporadic periods of long eyelid closure (followed by increased level of blink frequency) Yawn: some Body position: some situations with changing position e.g. stretching Body movements: some arms, legs, scratching, rubbing eyes ORS 2: Very sleepy, great effort to keep awake Blink: half-closed eyes, empty gaze Yawn: yes Body position: yes - change often, stretch, slumped, hanging Body movements: yes - e.g. head nodding

56 Subjective (self-reported) sleepiness scales KSS Karolinska sleepiness scale 9 levels 1 = extremely alert, 2 = very alert, 3 = alert, 4 = rather alert, 5 = neither alert nor sleepy, 6 = some signs of sleepiness, 7 = sleepy, but no effort to keep awake, 8 = sleepy, some effort to keep awake, 9 = very sleepy, great effort keeping awake, fighting sleep Self-reported sleepiness, CTU in Prague scale (Mashko et.al., 2016) 5 levels Sleepiness scale. Faculty of transportation Sciences CTU in Prague Score State description as perceived by driver 1 I feel fine/fresh & driving does not make me any problems. 2 I feel drowsy & driving does not make me any problems. 3 I feel drowsy & I notice some problems. 4 I feel very drowsy & I need excessively concentrate to drive correctly. 5 I experienced blackouts & losing of control over the car.

57 Self-reported sleepiness, CTU in Prague scale Self-rating scores provided by 5 subjects during sleep deprived state. p p2 p3 p4 p Self-evaluation scores of 5 subjects during 2 hours of driving (columns 1 to 9 correspond to the first hour, columns 10 to 18 correspond to the second hour). Scores from 1 to 5 are given per scale presented in Table 1. Light green cell colours (weak shading) represent lower score of drowsiness, blue colour (strong shading) represent higher score of drowsiness.