Comparison of selected off-the-shelf solutions for emotion recognition based on facial expressions

Transcription

1 Comparison of selected off-the-shelf solutions for emotion recognition based on facial expressions Grzegorz Brodny, Agata Kołakowska, Agnieszka Landowska, Mariusz Szwoch, Wioleta Szwoch, Michał R. Wróbel Gdańsk University of Technology, Gdańsk, Poland, Abstract The paper concerns accuracy of emotion recognition from facial expressions. As there are a couple of ready off-the-shelf solutions available in the market today, this study aims at practical evaluation of selected solutions in order to provide some insight into what potential buyers might expect. Two solutions were compared: FaceReader by Noldus and Xpress Engine by QuantumLab. The performed evaluation revealed that the recognition accuracies differ for photo and video input data and therefore solutions should be matched to the specificity of the application domain. Keywords emotion recognition; affective computing; facial expressions, FACS I. INTRODUCTION Affective computing, a research area willingly investigated in recent years, focuses on user emotions while interacting with computers and applications []. It refers to the problems of emotion recognition, reacting to them and influencing emotional states. Affective computing approach is more and more visible in applications used in various areas of our life, e.g. education [][], entertainment [][], software engineering [6], healthcare [7] and other domains [8]. The affective computing tools vary in many aspects. The most apparent one is the source of information taken into account to recognize users emotions. It may be visual [9], audio [], textual [], physiological [], or behavioral []. It is rather difficult to indicate the best approach in a general case. Some of these input channels appear to allow for more accurate emotion recognition than others. Some of them are known to be unobtrusive, whereas others are rather inconvenient for users. Facial expression analysis seems to be a compromise between high recognition accuracy and facility of convenient data acquisition. Moreover, it is the most natural method, as the face is usually the first, and often the only way humans show their emotional states. The aim of this article is to compare two applications dedicated to emotion recognition based on expression: FaceReader v. 6. by Noldus, Netherlands and Xpress Engine v..8. by QuantumLab, Poland. Both tools are available at Gdansk University of Technology as a part of Monitor stand [][]. Apart from evaluation of the two solutions, the comparison highlights different problems usually encountered while performing such evaluations. The paper is organized as follows: Section II presents some known solutions for emotion recognition based on facial expressions and a few projects aiming at benchmarking the recognition methods; Section III describes two tools selected for the study; Section IV presents research study plan; Section V contains recognition results obtained by both tools; Section VI is the analysis of the obtained results; Section VII summarizes the study and gives some ideas for the future work. The study might be interesting both for potential customers of the emotion recognition solutions and for researchers, who work on improving accuracies of existing emotion recognition methods. II. RELATED WORK A. recognition based on facial expressions There have been developed a number of research algorithms and methods to recognize emotions on the basis of image and video input. Most of them are research-based solutions [6], however there are also available several commercial off-the-shelf solutions. Most of them target at areas of marketing and usability testing. The most advanced solutions include (in alphabetical order): Affdex, Xpress Engine, Emotient, EmoVu, FaceReader, InSight and nviso. Algorithms implemented in these off-the-shelf solutions are mainly based on Facial Action Coding System (FACS). FACS is taxonomy of human facial movements (called action units, AU) by their appearance on the face. s are recognized as a combination of action units [7]. American psychologist Ekman noticed that some facial expressions corresponding to certain emotions are common for all the people independently of their gender, race, education, ethnicity, etc. He proposed the discrete emotional model using six universal emotions: happiness, surprise, anger, disgust, sadness and fear [8]. The facial expressions reflecting the Ekman s six basic emotions are shown in Fig.. Fig.. Sample of facial expressions characteristic for Ekman's six basic emotions This is a preprint of the paper presented at the 9th International Conference on Human System Interaction, Portsmouth, UK, July 6-8, 6, and published in the IEEE Xplore Digital Library DOI:.9/HSI

2 The off-the-shelf solutions usually provide a subset or a superset of Ekman's basic six emotions as an output. They also usually provide a value for neutral state. Affdex emotion recognition software works both as a stand-alone application and as online service, and provides frame-by-frame emotion metrics as an output. al state is described in the dimensional model (valence, attention, expressiveness) or as the discrete measure (smile, concentration, surprise and dislike) [9]. Emotient detects the 6 basic emotions (joy, surprise, anger, disgust, sadness and fear), and also contempt, frustration, and confusion. It also allows the detection of 9 of the Facial Action Units []. EmoVu API along with the basic emotional expressions, outputs complex features calculated from the basic ones []. InSight claims over 9% accuracy for neutral and 6 basic emotions []. nviso provides cloud service to measure emotional reactions of users. It uses FACS system, tracking the movement of facial muscles in real time []. Xpress Engine and FaceReader are described in more detail in Section III. According to our knowledge, the off-the-shelf solutions are rarely compared. Companies, that offer such solutions, are interested in sharing the best scenario results. On the other hand, researchers usually compare their own algorithms to the commercial one they have access to. B. Benchmarking emotion recognition methods This study is also based on and related to research, that aims at benchmarking emotion recognition algorithms. One of the most common approaches is to organize challenges colocated with main conferences and workshops in the field. There have been several events organized already, that aimed at comparison of emotion recognition algorithms. During the FERA challenges specific evaluation procedures for recognition algorithms were proposed []. Even further went Audio/Visual Challenge (AVEC) [] and Recognition in the Wild (EmotiW) [] events, which were organized in a form of competition between research teams around the world. In the first AVEC event was organized. It aimed at comparison of methods for automatic audio and visual emotion recognition. The competitors were challenged in measuring accuracy over four dimensions: activity, expectation, power and valence. The recognition was performed on the videos from the SEMAINE database. All videos were annotated by to 8 raters [6]. During the third AVEC challenge the number of dimensions was reduced to two: arousal and valence. The challenge used the audio-visual depressive language video database. All videos were annotated by raters []. The goal of the Recognition in the Wild (EmotiW) challenge was to provide a platform for researchers to benchmark the performance and accuracy of emotion recognition methods. The task was to classify sample audiovideo clips from the AFEW database into one of the seven categories: anger, disgust, fear, happiness, neutral, sadness and. Each clip was annotated by raters []. The Facial Expression Recognition and Analysis (FERA) challenge targets at facial expression recognition of on universal emotion classes presented on recordings taken by actors in lab conditions. Unlike the other challenges, the FERA goal was to estimate FACS Action Unit occurrence and intensity. Action Units were annotated frame-by-frame by a team of experts. The harmonic mean of recall and precision was selected as the AU occurrence measure. For comparison of AU intensity the Intraclass Correlation Coefficient was chosen []. To sum up, most challenges used the following method for emotion recognition algorithms benchmarking: first a set of photos or videos was chosen, then it was manually annotated by a number of raters and the algorithms were evaluated using some measures of consistency with manual annotations. In this study, we follow the same procedure. It is important to emphasis, that according to our best knowledge, no commercial off-the-shelf solutions took part in the challenges. III. DESCRIPTIVE ANALYSIS OF EVALUATED EMOTION RECOGNITION TOOLS Xpress Engine from QuantumLab, Poland is an application which allows to recognize and analyze emotions elicited by different stimuli [7]. The analysis can be performed both in real time as well as in off-line batch mode and is based on FACS model. Xpress Engine can detect five out of six Ekman s emotions: joy, disgust, surprise, sadness, anger, and additionally the neutral state. Xpress Engine recognizes also a subset of Action Units from FACS. As the output Xpress Engine provides vector of recognized emotions and AU for each frame. A sample screenshot from Xpress Engine is shown in Fig.. Face and eyes areas are marked on the live view (left). On the right, there is a normalized face in grey scale provided as well as estimation on the face rotation (small twoshaded window in the bottom right corner). Recognized emotions are visualized next to the live view window with a set of bars for each of the recognized emotional state. The recognized expressions are marked on the bars and the shaded area represents a notion of expression intensity. Fig.. QuantumLab Xpress Engine screenshot. The analysis of Xpress Engine interface highlights the most important issues in emotion recognition from facial expressions: recognition of face in the whole available scene, recognition of eyes within the face, recognition of expressions on normalized face image, elimination of the influence of face angle towards camera on affect recognition result. FaceReader from Noldus, Netherlands, similarly to Xpress Engine, can recognize emotions by analyzing user s facial expressions [8]. The application automatically analyzes six

3 Ekman s emotions: joy, disgust, surprise, sadness, anger, fear as well as neutral and, additionally, contempt. FaceReader analyzes a subset of commonly used Action Units. Facial expressions recognized by FaceReader can be represented as a value of valence and arousal in the circumplex model of emotions as well as the value of recognized emotions. A sample screenshot from FaceReader is shown in Fig.. Fig.. Noldus FaceReader screenshot (part of the main window). FaceReader software provides live analysis view, and recognized face is also marked with a rectangle shape. It is also possible to display additional masks on the face to get some insight into the recognition process. In the upper right corner, emotions are represented with bars as well as pie chart. Bottom chart visualizes valence dimension of the recognized emotional state. The interface is customizable and additional sub-windows may be displayed interchangeably. Both solutions might perform real-time or batch analysis and in the latter mode, results are provided as a set of emotion vectors recognized for each video frame. The data may be processed, then analyzed independently. Sample results from both solutions for the same video file are presented in Fig.. For visibility reasons, only three emotions series are visualized: anger, disgust, neutral. The values for other emotions were close to zero. The analysed video clip was originally annotated with anger label. The sample confirms observation, that anger is usually confused with disgust (both by recognition algorithms and manual annotators) as some action units are common for the two expressions. Noldus FaceReader solution seems to smooth the results for consecutive frames and calculates neutral emotion as a complement for recognized intensity of other emotions. Both algorithms agree in general (both recognize symptoms of anger and disgust), however they differ in intensity of the emotions, as well as the dominant emotion for frames. As the clip (and most clips in the annotated datasets) are usually assigned with one label only, some discretization procedure must be proposed to provide single label out of the emotion vectors series provided by the solutions. The procedure used in this study is described in Section IV. Fig.. An example output generated by the two compared solutions IV. STUDY DESIGN AND EXECUTION The aim of the presented study is to compare the effectiveness of two selected tools designed to recognize emotional states from facial expressions. A black-box approach has been adopted to make the comparison, i.e. the final output information on the emotions generated by both tools is the only aspect taken into account during the evaluation. Moreover, due to the type of information provided as annotations in the databases, discrete representation model has been chosen to describe emotional states. A. Selection of databases Facial expressions and emotions databases are widely used for training and evaluation of emotion recognition methods. Many such databases are available, some of them for free provided they are used for non-commercial purposes, such as research or education [9]. However they differ in terms of size, content and quality. Based on the criteria provided by Kolakowska et. al. [] two databases have been chosen: Extended Cohn-Kanade [] and MMI []. These two datasets are among the most popular and this means that they

4 are frequently used in emotion recognition algorithms benchmarking. One of the first attempt to develop a comprehensive database for facial expression analysis was Cohn-Kanade AUCoded Facial Expression Database, in which each sequence was labeled both by the emotion intended to be expressed as well as by observed facial movements using FACS [7]. Unfortunately, sequences were not verified against the real facial expressions they contained and the emotion labels referred to what expressions were requested rather than what could actually have been shown The extended version of this database, CK+, contains 9 sequences with more frames per sequence []. CK+ contains both posed as well as spontaneous facial expressions. In this version, validated emotion labels are provided together with FACS coding. Additional features are recognition results for facial feature tracking, action units and emotions. Although the CK+ database is very well prepared for use in many application fields, it contains only video sequences which can be limiting in some research. MMI Facial Expression Database contains over 9 video clips and still images, annotated using FACS. Additionally, 7 clips are labeled to indicate which of the six basic Ekman s emotions occurred. The MMI database contains 69 different subjects, both male and female, ranging in age from 9 to 6, having either European, African, Asian or South American ethnic background. The dataset is freely shared through a web-based application, which allows filtering and downloading selected recordings [], []. MMI is considered as a challenging database because the subjects express emotions in very different ways. Furthermore, some of them wear glasses, hat, have beards etc. []. All MMI recordings are stored as AVI files with standard codecs. All video clips were recorded in a controlled laboratory environment with homogeneous background without other people or objects visible. There are very good lighting conditions and recorded faces are even illuminated. B. Testing procedure for Cohn-Kanade database The first facial expression database used during tests was Cohn-Kanade, a standard-de-facto solution for testing emotion recognition algorithms. The methodology of photos selection for the study is described below. ) Selecting Cohn-Kanade database subset Cohn-Kanade database contains sequences of photos (selected frames) of certain emotion expressions, where the first photo in the sequence represents neutral state and the final one - the most intense emotional expression (fear, joy, disgust, surprise, sadness or anger). Intermediate photos in the sequence represent all stages of emotion intensity from neutral up to maximum state. For the purpose of the experiment a subset of photos was selected. In order to assure, that the photos are unambiguously labelled, intermediate states were excluded from the analysis. From each sequence, the first photo was selected with a label of neutral state and three up to five last photos in the sequence were selected with a label of the represented emotion. s representing fear were not included in the testing set. ) Performing automated emotion recognition on selected Cohn-Kanade database subset. Both Noldus FaceReader and QuantumLab Xpress Engine performed analysis on photos providing emotion vectors as an output. The assigned labels were then confronted with the Cohn-Kanade original labels of pictures. During the execution, all selected photos were successfully processed by Xpress Engine, while Noldus returned Unknown for a number of them (99 photos, 7% of the database). ) Visualising results as confusion matrix. The results are visualized as typical confusion matrix. Although no fear-labeled photos are analysed, the row and column for fear class was not excluded, as Face Reader software sometimes returns the label. Apart from recall and precision metrics for classes, overall accuracy is provided as a weighted average recall. C. Testing procedure for MMI database The second database used in tests was MMI Facial Expression Database. It was processed according to the following procedure: ) Selecting MMI database subset. From MMI database a subset of 7 video clips annotated with emotions was selected. Up to clips showing emotions: anger, joy, surprise, disgust, sadness plus neutral state were selected for manual annotation. Recordings with fear labels were excluded from the analysis, as Xpress Engine algorithm does not recognize this expression. ) Manual annotation of selected MMI subset Clips in MMI database are labeled with expressions, that the subjects were asked to show, therefore the labels accuracy might be low. Selected subset of video clips was manually annotated with 6 basic emotions (including fear) plus neutral state by 6 independent experts. was included in annotation, as some subjects perform fear expression instead of surprise. From 6 annotations fear was assigned times (,%). ) Setting consistency threshold and selection of final MMI subset for comparison procedure. Consistency of manual annotations varied among clipsonly 6 (6%) clips were annotated with % consistency, while some clips were assigned with different labels. Consistency varied for different emotional expressions as well. In this study kappa coefficient was chosen and consistency measure, as it is frequently used in the annotation consistency calculation []. Table I shows number of clips per class, given the consistency threshold as well as total kappa coefficient for the whole set after elimination of clips below threshold. The threshold for the extent of agreement for one clip was set to,6 (at least of 6 annotating experts had to be consistent to include clip in further analysis). As a result, smaller subset of 8 clips representing anger, joy, surprise, disgust, sadness, and neutral was selected to perform comparison of the algorithms.

5 Recall [%] Precision [%] Recall [%] Unknown CONFUSION MATRIX OBTAINED FOR PHOTOS FROM COHNKANADE DATASET RECOGNIZED BY FACEREADER Correctly recognized photos: 77.9% Tables IV and V present confusion matrices obtained after recognizing emotions presented on video clips from MMI database using Xpress Engine and FaceReader respectively. In contrast to photos, this time FaceReader turned out to be better with accuracy 6.%. Xpress Engine reached.%. The results are far from satisfying. Precision [%]. 8. Recall [%] CONFUSION MATRIX OBTAINED FOR VIDEO CLIPS FROM MMI DATASET RECOGNIZED BY XPRESS ENGINE TABLE IV. Particular accuracies vary among emotions. Both tools recognize joy and surprise the best. The worst results are obtained for anger and disgust in the case of FaceReader or anger and sadness in the case of Xpress Engine. However, in the case of Xpress Engine, the worst result is over 7%, which is still a high rate. Apart from anger and sadness, all other emotions are recognized by Xpress Engine with accuracies higher than 8%. is the only emotional state recognized better by FaceReader than by Xpress Engine. The confusion matrices let us analyze the errors made by the application. It can be noticed that anger is often recognized as neutral. In the case of Xpress Engine sadness is also incorrectly recognized as neutral in many cases recognized % Correctly photos: 9. 6 V. RESULTS Tables II and III present confusion matrices obtained after recognizing emotions presented on photos from Cohn-Kanade database using Xpress Engine and FaceReader, respectively. It can be seen that Xpress Engine achieved higher accuracy of 87.6%, whereas FaceReader 77.9%. However, there is an issue, which should be taken into account while comparing these results. It is the fact that Xpress Engine does not recognize fear, whereas FaceReader does. Although there were no fear examples in the test data, some examples have been recognized as fear by FaceReader. If these examples were excluded from the calculations, the result for FaceReader would be 78.76%, which is slightly better, but still much lower than the one obtained by Xpress Engine. TABLE III. 77 ) Performing automated emotion recognition on selected MMI Clips. Both Noldus FaceReader and QuantumLab Xpress Engine perform analysis on frame-by-frame basis and provide stream of emotion vectors as an output for each frame. The output therefore must be averaged or summed in order to obtain one dominant label for each clip. As MMI clips are recorded as a sequence of: neutral expression, emotion expression, neutral expression only middle one-third of frames was selected, each frame was assigned with vector of emotions from tested recognition algorithm each frame was assigned an Label that had maximum value within the vector for that frame, each clip was assigned an Label that was dominant for the middle one-third of frames (mode was calculated as maximum number of occurrences). ) Visualising results as confusion matrix. The results are presented in the same way as in the case of Cohn-Kanade database. Precision [%],89, Recognized as Kappa CONFUSION MATRIX OBTAINED FOR PHOTOS FROM COHNKANADE DATASET RECOGNIZED BY XPRESS ENGINE Recognized as TABLE II. Recognized as (.8; ] (.6; ] [; ] Extent of agreement CONSISTENCY OF MANUAL ANNOTATION OF MMI SUBSET All TABLE I recognized % Correctly videos:

6 Precision [%] Recall [%] CONFUSION MATRIX OBTAINED FOR VIDEO CLIPS FROM MMI DATASET RECOGNIZED BY FACEREADER Recognized as TABLE V recognized. 6.% Correctly videos: In the case of Xpress Engine joy and surprise are recognized with highest accuracies over 7%, disgust reaches 6.6%, other states show much lower rates. In the case of FaceReader the best results were achieved for disgust (7%). Accuracies for surprise and neutral state reached 6%, for anger and joy only slightly worse and for sadness the worst. Another issue worth noting is low precision in some cases, e.g. neutral for both applications. It means, that when the application returns neutral, the probability of correct decision is low. In other words, other emotional states are often confused with neutral. VI. SUMMARY OF RESULTS AND DISCUSSION In the comparison of affect recognition solutions based on facial expressions two off-the-shelf applications were explored using a black-box approach. Table VI presents summary results obtained for both databases using FaceReader and Xpress Engine. To sum up, the results obtained by the two applications in the case of photo database are rather high (87.6% and 77.9%). Those obtained for videos are much lower (.% and 6.%), but the number of emotions recognized was six, which means than even for videos the recognition accuracy was much better than random guessing. TABLE VI. SUMMARY RESULTS OBTAINED FOR PHOTO AND VIDEO DATABASES USING FACEREADER AND XPRESS ENGINE FaceReader Xpress Engine Database Precision [%] Recall [%] Precision [%] Recall [%] This study results may be formulated as follows: () key difference between Noldus FaceReader and QuantumLab Xpress Engine is the fear category, that is provided by the first one, and omitted by the second; () QuantumLab Xpress Engine solution performs slightly better on picture database with good recording conditions (illumination, angle); () Noldus Face Reader solution performs better on video database; () none of the solutions is clearly better, the accuracies and correctly recognized/confused emotional states are different. Finding the reasons behind the discrepancy in accuracies requires further research and is beyond this study scope, however seems an interesting direction for future works. We acknowledge that our approach to this study and analysis has some limitations, the main ones including a limited number and arbitral choice of databases for testing, limitations of chosen discretization methods and a quantitative black-box approach for results analysis. The choice of the databases is justified in Section IV, however we acknowledge, that more datasets would provide more valuable results. The databases we have chosen cover diversity in only one of the important characteristics: photo/film input. Another issue is how algorithms deal with conditions that are more natural and the presented research will be continued to address this issue. There are many different methods of discretization of emotional states from vectors of Ekman's six basic emotions values to single labels. The chosen discretization method is the simplest, straightforward approach and is adjusted to the characteristics of the tested datasets. The main limitation of the study is the quantitative blackbox approach chosen for the analysis of the results. The aim of the paper, however, was to quantify the accuracies of the algorithms with diverse databases and that approach was the most effective. However, some questions remain, including exploration of the reasons for the discrepancy of accuracy with photo/video data sets. Analysis of intermediate results (for example visualized in Figure ) suggests, that FaceReader uses smoothing based on neighboring frames and calculates neutral state as one minus the dominant emotion. However exploring this thesis might require a more qualitative whitebox approach. Although these results seem satisfying from the point of view of emotion recognition problem, it has to be highlighted, that these levels of accuracy would not be good enough in some practical applications. VII. CONCLUSIONS AND FUTURE WORKS In this paper, two off-the-shelf solutions for emotion recognition based on facial expressions were validated. Both FaceReader and Xpress Engine work well with video and still images database. FaceReader gives slightly better results for video recordings while Xpress Engine for still images. Although such difference does not let us clearly indicate a better solution, it may be an indication for using one of the programs according to the application field. Some advantage

7 of FaceReader is that it recognizes fear emotion, which is not possible with Xpress Engine. Both selected databases, MMI and CK+, contain recordings obtained in very good, controlled laboratory conditions: participants are correctly positioned in front of the monitor, the background and light conditions are proper and homogeneous. However, to be useful in real-life applications, the algorithms of emotion recognition should work with good accuracy also in not so ideal natural environment. One direction of our future work is to extend the validation tests on other video databases that meet these requirements. One of such databases is FEEDB that contains video clips of participants recorded in standard scenery of IT laboratory with uncontrolled mixed natural and fluorescent illumination [6]. Preliminary tests with FEEDB recordings proved that both applications achieve far lower recognition efficiency, but more tests and research are needed to find out the main factors that influence the results. Another direction of the work is to investigate more sophisticated methods for determining the actual user emotion taking into account highly dynamic changes of indicated emotions for particular frames. For this purpose, we plan to use low-pass time filtering and an expert voting system based on decision trees or neural networks. [9] [] [] [] [] [] [] [6] [7] ACKNOWLEDGMENT The research leading to these results has received funding from the Polish- Norwegian Research Programme operated by the National Centre for Research and Development under the Norwegian Financial Mechanism 9- in the frame of Project Contract No Pol-Nor/96/8/ as well as from DS Funds of Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology. The authors thank Michał Witkowicz for data preprocessing. [8] REFERENCES [] [] [] [] [] [] [6] [7] [8] R. Picard, Affective computing: from laughter to IEEE, IEEE Trans. Affective Computing (), pp. -7,. G. Tsoulouhas, D. Georgiou, A. Karakos A, Detection of learner s affective state based on mouse movements, Journal of Computing (),, pp A. Landowska, "Affective learning manifesto years later", th European Conference on e-learning (ECEL), Aalborg Univ, Copenhagen, Denmark, October, pp M. Szwoch and W. Szwoch, recognition for affect aware video games, 6th International Conference on Image Processing & Communications, Springer-Verlag,. A. Landowska and M.R. Wrobel, Affective reactions to playing digital games, 8th International Conference on Human System Interaction, Warsaw, Poland, pp. 6-7,. M.R. Wróbel, s in the software development process, 6th International Conference on Human System Interaction, Gdańsk, Poland,. S. Tivatansakul, G. Chalumporn, S. Puangpontip, Y. Kankanokkul, T. Achalaku, and M. Ohkura, Healthcare system focusing on emotional aspect using augmented reality: emotion detection by facial expression, Advances in Human Aspects of Healthcare, pp. 7-8,. A. Kolakowska, A. Landowska, M.Szwoch, W. Szwoch, and M.R. Wróbel, recognition and its applications, Human-Computer [9] [] [] [] [] [] [6] [7] [8] Systems Interaction: Backgrounds and Applications, Advances in Intelligent Systems and Computing, vol., pp. -6,. H. Gunes and M. Piccardi, Affect recognition from face and body: early fusion vs. late fusion, IEEE International Conference on Systems, Man and Cybernetics, vol., pp. 7,. B. Schuller, M. Lang, and G. Rigoll, Multimodal emotion recognition in audiovisual communication, Proceedings of IEEE International Conference on Multimedia and Expo, ICME, Lausanne,. A.J. Gill, R.M. French, D. Gergle, and J. Oberlander, Identifying emotional characteristics from short blog texts, th Annual Conference of the Cognitive Science Society, pp. 7-, 8. W. Szwoch, Using physiological signals for emotion recognition, 6th International Conference on Human System Interaction, Gdańsk, Poland,. A. Kołakowska, A review of emotion recognition methods based on keystroke dynamics and mouse movements, 6th International Conference on Human System Interaction, Gdańsk, Poland,. A. Landowska, monitoring - verification of physiological measurement procedures, Metrology and Measurement Systems, vol. (), pp. 8-88,. A. Landowska, " monitor - concept, construction and lessons learned", Federated Conf.on Computer Science and Information Systems, IEEE, pp.7-8,. A. Dhall, O. V Ramana Murthy, R. Goecke, J. Joshi, and T. Gedeon, and image based emotion recognition challenges in the wild: Emotiw, ACM International Conference on Multimodal Interaction, 6,. P. Ekman, W.V.Friesen and J.C.Hager Facial action coding system, A Human Face,. P. Ekman and W. V Friesen, Constants across cultures in the face and emotion, Journal of personality and social psychology, vol. 7(), pp. -9, 97. Affectiva, White paper: Exploring the emotion classifiers behind Affdex facial coding, Exploring_Affdex_Classifiers.p df, accessed..6. Bența Kuderna-Iulian and Vaida Mircea-Florin, Towards real-life facial expression recognition systems, Advances in Electrical and Computer Engineering, vol. (), pp. 9,. L. Liu, D. Preot, and L. Ungar, Analyzing personality through social media profile picture choice, The AAAI Digital Library, 6. A. Dinculescu, C. Vizitiu, A. Nistorescu, M. Marin, and A. Vizitiu, Novel approach to face expression analysis in determining emotional valence and intensity with benefit for human space flight studies, EHealth and Bioengineering Conference (EHB), pp.,. M. F. Valstar, T. Almaev, J. M. Girard, G. McKeown, M. Mehu, L. Yin, M. Pantic, and J. F. Cohn, FERA - second Facial Expression Recognition and Analysis challenge, th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 6, pp. 8,. M. Valstar, B. Schuller, K. Smith, F. Eyben, B. Jiang, S. Bilakhia, S. Schnieder, R. Cowie, and M. Pantic, AVEC - The Continuous Audio/Visual and Depression Recognition Challenge, rd ACM international workshop on Audio/visual emotion challenge AVEC, pp.,. A. Dhall, R. Goecke, J. Joshi, and T. Gedeon, and Image based Recognition Challenges in the Wild : EmotiW, ACM International Conference on Multimodal Interaction ICMI, pp. 6,. M. Valstar, F. Eyben, G. Mckeown, R. Cowie, and M. Pantic, AVEC The First International audio/visual emotion challenge, th International Conference on Affective Computing and Intelligent Interaction, pp. -,, Springer Verlag,. Xpress Engine solution description, accessed on..6. FaceReader solution description, accessed on..6.

8 [9] G. Castaneda and T. M. Khoshgoftaar, A survey of D face databases, IEEE International Conference on Information Reuse and Integration, pp. 9,. [] A. Kolakowska, A. Landowska, M. Szwoch, W. Szwoch, and M. R. Wrobel, Evaluation criteria for affect-annotated databases, Beyond Databases, Architectures and Structures, vol., Springer International Publishing, pp. 8 97,. [] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression, IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW, pp. 9,. [] M. F. Valstar and M. Pantic, Induced disgust, happiness and surprise: an addition to the MMI facial expression database, Int. Conf. Lang. Resour. Eval. Work. Emot., pp. 6 7,. [] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, Web-based database for facial expression analysis, IEEE International Conference on Multimedia and Expo, pp. 7,. [] L. Zhong, Q. Liu, P. Yang, J. Huang, and D. N. Metaxas, Learning multiscale active facial patches for expression analysis, IEEE Trans on Cybernetics, vol. (8), pp. 99-,. [] J. L. Fleiss, Measuring nominal scale agreement among many raters, Psychological Bulletin, vol. 76(), pp. 79-8, 97. [6] M. Szwoch, On Facial Expressions and s RGB-D Database, th International Conference, BDAS, Springer Verlag,.