Support Vector Machine Classification and Psychophysiological Evaluation of Mental Workload and Engagement of Intuition- and Analysis-Inducing Tasks

Transcription

1 Support Vector Machine Classification and Psychophysiological Evaluation of Mental Workload and Engagement of Intuition- and Analysis-Inducing Tasks Presenter: Joseph Nuamah Department of Industrial and Systems Engineering Advisor: Younho Seong March 3, 2017 Presenter: Joseph Nuamah March 3, / 34

2 1 Outline 2 Introduction 3 EEG 4 Aim 5 Hypotheses 6 Methodology 7 Results 8 Discussion and Conclusion Presenter: Joseph Nuamah March 3, / 34

3 Introduction: Human Factors(HF) Issues in Autonomous Vehicles(AV) Appropriate Levels of Automation Human operator in-the-loop for failure mode operations Fail-safe mechanisms into AV Usefulness of UV interfaces Presenter: Joseph Nuamah March 3, / 34

4 Introduction: Problems with Improper Design of Automation Increased monitoring demands Cognitive overload Mis-calibration of trust in automation Inability to resume manual control Loss of situation awareness Degraded manual skills due to lack of practice Presenter: Joseph Nuamah March 3, / 34

5 Introduction: System Monitoring Increased UV operator s role of monitoring and supervising automation Increased monitoring requirements add to cognitive load Vigilance - Failure detection worse under passive monitoring than under active control (Wickens & Kessel, 1980) UVs likely to contain more displays and instruments: Camera-fed screens Screens following up flight plan Instruments indicating intact communication requirements between Air Traffic Control operator and UV operator Presenter: Joseph Nuamah March 3, / 34

6 Introduction: Human Supervisory Control Human interactions with environment mediated by technological interfaces Supervisory control - operators oversee automated process, and continuously determine basis need to re-enter control loop Ongoing assessment based on comparison of actual and intended system performance UV operator in supervisory role requires information about target parameters decides how automation should proceed to achieve targets communicate appropriate instructions monitor process to ensure commands are understood and executed Components of Human Supervisory Control System: Human operator Interface Automation Presenter: Joseph Nuamah March 3, / 34

7 Introduction: Decision Making Design of interface requires understanding of judgment characteristics that they are to support effect of design on operator judgment Task characteristics play important role in determining cognitive mode likely to be used Higher correspondence between task characteristics and cognitive characteristics correlate with operator s judgment accuracy Dual process theories of intuition and analysis used to explicate human cognitive system Presenter: Joseph Nuamah March 3, / 34

8 Introduction: Decision Making (cont d) Table: Attributes of Intuition and Analysis Cognition (adapted from Evans & Stanovich, 2013) Intuition Analysis Does not require working memory Requires working memory Fast Slow High capacity Capacity limited Parallel Serial Nonconscious Conscious Automatic Controlled Holistic Analytic Relatively undemanding of cognitive capacity Demanding of cognitive capacity Experience-based decision making Consequential decision making Table: Inducement of Intuition and Analysis by Task Conditions (adapted from Hammond et al., 1987) Task Characteristic Intuition-Inducing State of Task Characteristic Analysis-Inducing State of Task Characteristic Number of cues Large (>5) Small Measurement of cues Perceptual measurement Objective, reliable measurement Distribution of cue values Continuous highly variable distribution Unknown distribution; cues are dichotomous; values are discrete Redundancy among cues High redundancy Low redundancy Decomposition of task Low High Availability of organizing principle Unavailable Available Degree of certainty in task Low certainty High certainty Time period Brief Long Presenter: Joseph Nuamah March 3, / 34

9 Introduction: Decision Making (cont d) Behavioral and subjective traditionally measures used to measure judgment and decision making performance May not produce much information on the operator s state Physiological measurements may be used Continuously available and collection does not interfere with operator s task performance Measures range from blood flow or neural activity in brain to heart rate variability and eye movements Methods include: Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (FNIRs), etc Skin conductance, cardiovascular responses, muscle activity, pupil diameter, eye blinks,eye movements, etc Presenter: Joseph Nuamah March 3, / 34

10 EEG EEG signals represent summed postsynaptic potentials of neurons firing a rate of milliseconds Graph of time varying voltage difference between active electrode attached to scalp and reference electrode Table: Lobes and corresponding electrode label Lobe Electrode Frontal F Temporal T Central C Parietal P Occipital O Presenter: Joseph Nuamah March 3, / 34

11 EEG (cont d) Figure: Time domain EEG signal Figure: Waveform showing several ERP components Table: Lobes and corresponding electrode label Frequency Band (Hz) Associated Tasks & Behaviors Delta (0.1-3) Lethargic, not moving, not attentive Theta (4-8) Creative, intuitive, distracted, unfocused Alpha (8-12) Meditation, no action Beta (12-30) Mental activity Gamma (>30) High-level information Figure: Main EEG waves Presenter: Joseph Nuamah March 3, / 34

12 EEG Indexes Spectral composition of EEG changes in response to changes in task difficulty or level of alertness Alpha, theta, beta all related to task engagement Task Engagement Index (TEI) is given by beta power alpha power + theta power Task Load Index (TLI) is given by f rontal midline theta parietal alpha Presenter: Joseph Nuamah March 3, / 34

13 Aim Employ TLI to provide insight into cognitive load Employ TEI to provide insight into engagement Employ SVM to discriminate EEG signals recorded during execution of intuition-inducing and analysis-inducing tasks Employ objective measures (reaction time and percent correct), and subjective measure (NASA-Task Load Index) to validate objective EEG measures (TLI and TEI) Presenter: Joseph Nuamah March 3, / 34

14 Hypotheses Engagement required for analysis-inducing tasks would be different from that required for intuition-inducing tasks Mental effort required for analysis-inducing tasks would be different from that required for intuition-inducing tasks Presenter: Joseph Nuamah March 3, / 34

15 Methodology: Materials and Method Participants: Six participants (1 female, 5 males) Ages between 18 and 35 years All right-handed Normal vision No history of neuropsychiatric disorders Equipment: g.hiamp-256 channel biosignal amplifier g.gammacap Electrode Type: AgCl Active electrode connector box comes with 64 channels g.recorder used to record the EEG signals Presentation Software for stimuli delivery Presenter: Joseph Nuamah March 3, / 34

16 Methodology: Stimuli Baseline: Participants were instructed to relax and fixate on a blank screen for 60 s Intuition-inducing task: For each stimulus, two objects presented: fixation on left, and flashing face on right Participants were instructed to press LEFT mouse button if they thought face on RIGHT was a happy face, or press RIGHT mouse button if they thought face on RIGHT was face of someone who was afraid Stimuli taken from FACE database established by Ebner et al. (2010) Stimulus duration was approx. 6 s Inter Stimulus Time (ISI) was approx. 2 s Two blocks, each containing 30 trials Presenter: Joseph Nuamah March 3, / 34

17 Methodology: Stimuli (cont d) Analysis-inducing task: For each stimulus, two multiplications were presented Participants instructed to determine which of two multiplications was larger Participants instructed to press LEFT mouse button if they thought multiplication on LEFT was larger or press RIGHT mouse button if Presenter: Joseph Nuamah March 3, / 34

18 Methodology: NASA-TLX Subjective workload assessment tool Overall workload score based on weighted average of ratings on six subscales: Mental Demand Physical Demand Temporal Demand Performance Effort Frustration Presenter: Joseph Nuamah March 3, / 34

19 Methodology: Procedure Sign informed consent and complete demographic questionnaire Fit g.gammacap on scalp: 20 electrodes used, ear lobe as reference Calibrate electrode impedance Present experimental conditions Record Response time (RT) Complete NASA-TLX questionnaire Presenter: Joseph Nuamah March 3, / 34

20 Methodology: Signal Preprocesssing Raw EEG signals recorded at sampling rate of 256 Hz with Butterworth filter (0.01Hz high pass - 100Hz low pass) Notch filter with 60 Hz cutoff frequency to remove line noise Data re-referenced to average EEG epochs time-locked to stimulus presentation Data with amplitudes outside of range of -50 µv to +50 µv rejected Independent component analysis (ICA) correct EEG data contaminated by signals of non-neural origin SASICA software used to reject artifact independent components before EEG data analysis Presenter: Joseph Nuamah March 3, / 34

21 Methodology: Signal Preprocesssing Presenter: Joseph Nuamah March 3, / 34

22 Methodology: Signal Preprocesssing Data submitted to ICA are EEG channel recordings arranged in a matrix of n channels (rows) by t time points (columns) data values ICA performs blind source separation of X with assumption that source time courses (U) are independent ICA finds component umixing matrix (W) that when multiplied by original data (X), yields matrix (U) of IC time courses U = W X X = W 1 U W 1 is n by n component mixing matrix whose columns contain relative weights with which component projects to each scalp channel Portion of original data X forms the i th IC(X i ) is (outer) product of two vectors: i th column of W and i th row of U Presenter: Joseph Nuamah March 3, / 34

23 Methodology: Signal Processing Parametric approach - assume a model for EEG signals and estimate parameters of model AutoRegressive-Moving-Average (ARMA) Nonparametric approach - do not require a model of signal Fast Fourier Transform (FFT) Assumes signals are stationary Short-Time Fourier Transform (STFT) Allows for depiction of nonstationary signals as stationary ones by use of window function Limitation - window too narrow implies poor frequency resolution, window too wide implies imprecise time localization Wavelet Transform (WT) Presenter: Joseph Nuamah March 3, / 34

24 Methodology: Signal Processing Wavelet - waveform of effectively limited duration, has average value of zero Wavelet analysis-breaking up of a signal into shifted and scaled versions of original (or mother) wavelet Two types: Continuous Wavelet Transforms (CWT) and Discrete Wavelet Transform (DWT) CWT defined as sum over all time of a signal x(t) multiplied by scaled, shifted versions of the wavelet function ψ C(scale, location) = x(t)ψ(scale, location)dt Two approaches differ in how they discretize scale parameter DWT choose scales and positions based on powers of 2, CWT uses exponential scales with a base smaller than 2 Presenter: Joseph Nuamah March 3, / 34

25 Methodology: Signal Processing CWT expressed as C(a, b) = 1 a ( ) t b x(t)ψ dt a where a, b R, a 0 a is the scale parameter, b is the location parameter, ψ(t) is the mother wavelet 1 a used to normalize energy such that it stays as same level for different values of a and b Choice of mother wavelet depends on kind of features to be extracted CWT scales and frequency are related by where F a = F c a. a is a scale is the sampling period (1/256) F a is the pseudo-frequency corresponding to the scale a, in Hz center frequency of a Morlet wavelet is Presenter: Joseph Nuamah March 3, / 34

26 Methodology: Signal Processing where Coefficients of CWT for scale of 1.5 to 80 with a scale step of 0.1 Computed geometric mean energy of wavelet coefficients of each scale using E j = 1 n x i 2, j = 1,..., 786 n i=1 x i s are computed coefficients of signal at each scale n is window size 786 is total number of scales Presenter: Joseph Nuamah March 3, / 34

27 Methodology: Signal Processing Table: CWT SCALE RANGE AND CORRESPONDING PSEUDO-FREQUENCY AND EEG BAND Scale Range PseudoFrequency (Hz) EEG Band Gamma Beta Alpha Theta Delta Corresponding scale ranges for delta, theta, alpha, beta, and gamma bands used in present study are shown in Table Mean value of geometric means at each scale gives their corresponding absolute energy, E band For an EEG signal, total energy E t across all five bands is given by Relative wavelet energy is computed as E t = E delta + E theta + E alpha + E beta + E gamma ρ band = E band E t Computed five relative wavelet energy features -resulted in 100 features (5 features x 20 channels) per trial per task Presenter: Joseph Nuamah March 3, / 34

28 Methodology: SVM Classification Performed all classifications offline Designed six separate SVMs to classify two cognitive tasks for each participant SVM was tested with Radial Bias Function (RBF) kernels 10-fold cross-validation to find the best C and γ Presenter: Joseph Nuamah March 3, / 34

29 Methodology: Block Diagram Presenter: Joseph Nuamah March 3, / 34

30 Results Multivariate Analysis of Variance (MANOVA) technique used Dependent variables: mean reaction time (RT), mean NASA-TLX, mean percent correct, mean TLI across all channels for all participants, mean TEI across all channels for all participant Results there was a significant overall treatment effect on NASA-TLX, Percent Correct and RT when analyzed simultaneously Wilk s Λ = , F(3,8) = , p- value < Indicates at least one effect of three task types on NASA TLX, Percent Correct and RT is different from others Univariate ANOVA to study effects of the two task levels on each dependent variable NASA TLX-enough evidence to conclude that the two different treatment levels did not have the same treatment effect on TLI, F(1,10)= , p- value < Percent Correct-enough evidence to conclude that the two different treatment levels did not have same treatment effect on TLI, F (1,10) = 27.86, p-value < RT-enough evidence to conclude that the two different treatment levels did not have same treatment effect on TLI, F(1,10) = 28.32, p-value < Presenter: Joseph Nuamah March 3, / 34

31 Results (cont d) Table: DESCRIPTIVE STATISTICS FOR EFFECT OF TASK TYPE ON NASA TLX, PERCENT CORRECT, AND RT Task Type Dependent Variable Mean NASA-TLX ± 4.16 Analysis-Inducing Percent Correct ± 3.07 RT ± NASA TLX ± Intuition-Inducing Percent Correct ± 1.04 RT ± Presenter: Joseph Nuamah March 3, / 34

32 Results (cont d) Table: DESCRIPTIVE STATISTICS FOR EFFECT OF TASK TYPE ON TLI AND TEI Task Type Dependent Variable Mean Analysis-Inducing TLI 5.23 ± 1.73 TEI 0.47 ± 0.58 Intuition-Inducing TLI 2.42 ± 0.38 TEI 0.58 ± 0.75 Baseline TLI 3.80 ± 1.61 TEI 0.49 ± 0.58 Presenter: Joseph Nuamah March 3, / 34

33 Results (cont d) Presenter: Joseph Nuamah March 3, / 34

34 Discussion and Conclusion Results from statistical analysis were consistent with hypothesis TEI for each participant across all 20 EEG channels revealed that TEI for baseline was generally higher than intuition-inducing and analysis-inducing tasks for all participants Higher value of TEI generated by intuition-inducing tasks in part as result of flashing nature of stimuli presented Negative correlation found between TLI and TEI - TLI measures mental workload, while TEI measures alertness and engagement Average classification accuracy of % Analysis-inducing tasks appear to impose higher cognitive loads than intuition-inducing tasks SVM may be employed to classify EEG signals recorded during execution of intuitionand analysis-inducing tasks, and by extension high and low cognitive loads Presenter: Joseph Nuamah March 3, / 34