Classifying Facial Pain Expressions

Transcription

1 Classifying Facial Pain Expressions Individual Classifiers vs. Global Classifiers Michael Siebers 1, Miriam Kunz 2, Stefan Lautenbacher 2, and Ute Schmid 1 1 Faculty Information Systems and Applied Computer Science, University of Bamberg, Germany 2 Physiological Psychology, University of Bamberg, Germany Abstract. Pain is a highly affective state that is accompanied by a facial expression. In this paper we compare different classifiers on their possibilities to classify pain from facial point data. Furthermore we investigate the need for training classifiers on each subject s data individually. We show that most used classifiers are suitable for the classifying facial pain expressions. The second issue cannot be finally decided. 1 Motivation and Introduction Pain is an highly unpleasant feeling everybody is familiar with. Humans have an urge to communicate pain to others. This is done both verbally the typical ouch and nonverbally. Other persons react on those pain messages, e.g. with pity by providing help. Important for nonverbal communicating pain is the facial expression. Especially for those individuals that are not able to communicate pain verbally (e.g. patients with dementia) the facial expression of pain can serve as a valid communication channel of pain [1, 2]. This finding can be valueable for the clinical context where individualized classifiers for pain can support relatives and nursing staff in recognizing pain states of patients[3]. This pilot study tries to find possibilities for deciding whether a person experiences pain using facial images. To be precise, this study tries to answer two questions. First, which classifiers are suitable to predict pain by facial features. Second, is a global classifier, trained on different persons, sufficient or must individualized classifiers be trained for each person as psychological studies [1, 2] suggest. Since this is a pilot study we are only handling a limited number of data records. The work reported in this paper is mainly explorative concerning the general investigation of suitable learning approaches, appropriate feature selection, and efficient preprocessing of data. Most importantly, we aimed at getting first support for the psychological hypothesis that pain classification must be dealt with individual classifiers and that general classifiers as often used for classification of emotions as happiness [4] are significantly less reliable.

2 2 Data Acquisition In order to learn classifiers we needed pain-annotated facial data. These were derived from videos obtained during a previously conducted psychophysiological study and stills of non-painful and painful events were extracted. On those images facial features were annotated. Based on these features relational measures were calculated. 2.1 Psychophysiological Study In this study, pressure stimuli of various intensities were applied to the upper edge of the trapezius muscle using a pressure Algometer (SOMETIC). Each subject received 40 stimuli, half-and-half painful and not painful. As the first stimulus a pressure of 5 kg was applied. This stimulus was repeated with increasing weight until the subject showed a pain face or the maximum of 8 kg was reached. This target intensity was used as the painful stimulus, whereas 1 kg was used to induce non-painful pressure sensations. A total of 30 subjects took part in the study. Each of them was video taped. In the background a LED light lit up while pressure was applied. In addition to the video recordings, the subjective pain ratings were assessed on a Visual Analogue Scale 3. Furthermore, the pressure intensity and expert s ratings on whether subjects displayed pain-typical facial expressions (this was done by M. Kunz who is a certified coder for facial expressions) were recorded for each stimulus application. As this study was not initially conducted to be used for automated facial pain detection, it is no surprise that we came across some problems concerning image quality. The main one was that some subjects hair was hiding parts of their faces. 2.2 Still Extraction Since we wanted to evaluate the classification performances of different classifiers, we decided to use ideal data only. Therfore among all subjects only those showing the most prototypic pain faces were chosen for further consideration. Six person showed appropriate mimics. From their videos ten stills per stimulus were extracted. Stills were taken from all 20 non-pain stimuli and all pain stimuli showing a pain face. All stills were taken from within the first second after facial expression onset. This resulted in a total of 1200 non-pain and 880 pain images. From further study one subject was excluded. He showed a pain face only in six of the 20 pain stimuli others at least twice as much. Additionally he was the only male among the remaining subjects. 3 A visual analogue scale (VAS) is a scale where values are stated on a continuum ranging from one extreme to another. In this study it is implemented as percentage value.

3 Fig. 1. Annotated feature points and shapes. 2.3 Facial Feature Annotation On the images we marked 58 points as seen in Fig. 1. Those formed the face s contour (13 points), mouth (8 points), nose (11 points), eyes (8 points each) and eye brows (5 points each). We chose those points with FACS [5] in mind. However, we didn t use FACS directly, as we are in doubt whether it is suited to map facial pain expressions. For the annotation process we used the AM Tools of Tim Cootes [6]. As the images hardly displayed variation only the first three images of each stimulus were used. Few stimuli had to be excluded as the face was not completely on the images. So the final data set consisted of 534 records (294 non-pain; 240 pain) of five female subjects. To eliminate the size and the position of the face within the image the x- and y-coordinates of the points were standardized to a range of [0, 1]. 2.4 Relational Measures To the pure data record additional attributes were added. Distances considered as relevant were calculated and added to the data record. width and height of the mouth widths and heights of both eyes span from the tip to the root of the nose distance between the eye brows distances between eye and eye brow distance between mouth and nose All distances can be seen in Fig. 2. Moreover we added the angle between the mouth corners and the mouth centre.

4 Fig. 2. Calculated distances and angles It might be that these distances scale proportionally, or it might not be. To survey this we included proportions into the data set. ratio of mouth height to mouth width ratio of eye height to eye width (both sides) [called: eye ratio] ratio of left eye width to right eye width ratio of left eye height to right eye height ratio of left eye ratio to right eye ratio It is understood that the pain face is not the usual face but a distortion of the neutral face. Taking this into account we averaged the distances, angles and ratios of the non-pain data records and added for every record the ratios of those attribute to their means. As the deformation of the face is potentially highly individual we also added the ratios of those attributes to their means of the considered subject s non-pain data. Of course all mentioned relational measures are implied in the point data alone. However they would be inaccessible for some classifiers, e.g. Decision Trees (see Sect. 3.1). Hence we decided to include those measures in an explicit form. 3 Experiments Based on the data set with 534 entries and 178 attributes (including image id, person number and class) classifiers were trained. 3.1 Classifiers In the following the used classifier are briefly described. For further reading see [7] or [8](Support Vector Machines).

5 Decision Trees classify an example according to a set of tree structured if-thenrules. Each inner node of the tree denotes an attribute and branches according to its values. The Decision Tree classifier is the only symbolic classifier we used. We considered using it as the constructed decision model is human understandable and thus has an explaining element. For learning this trees we used ID3Numerical, a modification of Quinlan s ID3 [9] which can handle numerical attributes. Naive Bayes is a statistical classifier that is based on Bayes Law. It estimates the probability P (c x). For this purpose it assumes that all attributes are statistical independent (thus naive). Support Vector Machines try to find a hyperplane probably in a higher space that separates the classes. For classification only the data records closest to the hyperplane are used the Support Vectors. k-nearest Neighbours (knn) considers each data record as points in IR n. An unseen instance is now classified searching the k nearest points in the training example space and assigning their mode class. All training examples are kept in the feature space. Therefor knn is called a lazy learner. The Perceptron considers each data record as a vector of real-valued inputs (x 1,..., x n ). Those are weighted with (w 0,..., w n ). For classification it calculates the linear combination n i=0 w ix i, where x 0 is always 1. If the result is greater than 0 it returns positive, negative otherwise. Neural Networks use a graph of sigmoid units to calculate the desired class. Sigmoid units are similar to perceptrons except that they use a sigmoidal output. This output is serving as input for the next unit. In this study we used a linear and acyclic network. Classification by Regression trains a regression model for each class. The class with the higher predicted value is then selected. As base regression model we used linear regression. Linear regression tries to find the linear function that predicts the examples best. 3.2 Procedure For each classifier the learning generally consisted of two phases. First its optimal parameters were obtained then its performance was measured in an crossvalidation. As knn suffers from curse of dimensionality, the attributes were weighted for this classifier before parameter optimization. Cross-Validation was done using 10 partitions. The partitions were drawn according to single data records by means of stratified sampling 4. 4 Stratified exampling means randomly assigning data records to folds while trying to presume class distribution.

6 Parameters were optimized via a systematic grid search. Parameter combinations were systematically evaluated using 10-folded cross-validations. The combination which performed best was selected. For the Neural Network no systematic approach was chosen. Taking the long training times and the size of the parameter space into account we decided on an evolutionary technique. Attribute Weighting was performed using forward weighting. Initially every attribute was assigned a weight of 0. Each attribute was then independently weighted using a linear search. Attribute Selection was used to overcome Naive Bayes assumptions that attributes are statistical independent. Carried out as forward selection, an initial population is created one individual per attribute. Then further attributes are added to the best ones as long as performance increases. This proceedings were carried out with the whole data set (global) and once for each subject using only its data (individual classifiers). We ran the experiments with RapidMiner 5 on a Fujitsu Siemens Computers LIFEBOOK T Series (Intel c Core TM 2 Duo P8400, 2 GB) and a Fujitsu Siemens Computers Esprimo (Intel R Pentium R GHz, 1 GB). 4 Results The results of the different classifiers are shown in Tab. 1. Obviously most classifiers are suitable for this task. Table 1. Experiments results global individual Decision Tree Support Vector Machine Regression Perceptron a b Neural Network c d Naive Bayes k-nearest Neighbours a Parameter Optimization canceled after 19 days. Best interim result is displayed. b Parameters not optimized due to time constraints. c Parameter Optimization canceled after 10 days. Using not optimal parameters. d Parameters not optimized due to time constraints. 5

7 4.1 Classifier Performance Apparently Support Vector Machines and k-nearest Neighbours perform best. Unfortunately it is not possible to build an exact ranking. Due to the number of subjects, no significance tests were made. Decision Tree, Support Vector Machine, Regression, Naive Bayes and k-nearest Neighbours perform similar as adults classifying clinical pain videos[10]. No detailed statement about the Neural Network can be made. The global parameter optimization crashed after 10 days. Since no interim results were available performance estimation was done with manually chosen parameters. A repetition of the parameter optimization was not possible due to time limitations. The detail plot in Fig. 3 shows that the data is probably not linear separable. This explains the bad performance of the Perceptron classifier. It performs even worse than guessing 6. Fig. 3. Detail plot of the used data set. The topmost attributes of the global decision tree model were used as axis. 4.2 Global vs. Individual Classification For most classifiers the individual approach seems to deliver better results. But due to the small number of subjects also here no detailed comparison is possible. 6 Guessing probability: for global classification, for individual classification

8 5 Conclusion and Future Work We showed that many classifiers are suitable for predicting pain by facial expression. Global and individual classifiers perform nearly equally. But due to the low subject number and since we only selected those subjects that showed prototypical facial pain displays[11] it is by now questionable if these findings can be generalized. Currently we are able to decide if a shown face is prototypical for pain or not. However, the true question is whether the person whose face is depicted is experiencing pain. Therefore we will deal with predicting the self-reported VAS values in further work. Additionally we will tackle the issue of mixing up different emotions, first trying to distinguish between pain and disgust. Therefore we will use more subjects for the initial psychological study. Further studies should also address a broader variety of subjects for example regarding age or gender. Acknowledgements We thank Simone Burkhardt and Ina Schulz for the data collection and picture extraction. The study was supported by the Dr. Werner Jackstädt-Stiftung. References 1. Kunz, M., Mylius, V., Scharmann, S., Schepelman, K., Lautenbacher, S.: Influence of dementia on multiple components of pain. European Journal of Pain 13 (2009) Kunz, M., Scharmann, S., Hemmeter, U., Schepelman, K., Lautenbacher, S.: The facial expression of pain in patients with dementia. PAIN 133 (2007) Salomon, P.E., Prkachin, K.M., Farewell, V.: Enhancing sensitivity to facial expression of pain. PAIN 71 (1997) Strupp, S., Schmitz, N., Berns, K.: Visual-Based Emotion Detection for Natural Man-Machine Interaction. In Dengel, A.R., Berns, K., Breuel, T.M., Bomarius, F., Roth-Berghofer, T.R., eds.: KI 2008: advances in artificial intelligence: 31st Annual German Conference on AI, KI 2008, Kaiserslautern, Germany, September 23-26, 2008 ; proceedings. Volume 5243 of LNAI., Berlin, Springer (2008) Ekman, P., Friesen, W.V.: Facial action coding system. Consulting Psychologists Press, Palo Alto, Calif. (1978) 6. Cootes, T.F.: Am Tools. timothy.f.cootes/software/am tools doc/index.html 7. Mitchell, T.M.: Machine learning. McGraw-Hill, Boston, Mass. (1997) 8. Bishop, C.M.: Pattern Recognition and Machine Learning. 1. aufl. edn. Springer Science + Business Media LLC, New York, NY (2006) 9. Quinlan, J.R.: Induction of decision trees. Machine Learning 1(1) (1986) Deyo, K.S., Prkachin, K.M., Mercer, S.R.: Development of sensitivity to facial expression of pain. PAIN 107(1-2) (2004/1) Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. PAIN 51 (1992)