The Analysis of Learner s Concentration by Facial Expression Changes & Movements

Transcription

1 The Analysis of Learner s Concentration by Facial Expression Changes & Movements Seunghui Cha 1 and Wookhyun Kim 2 1 Ph.D Student, Department of Computer Engineering, Yeungnam University, Gyeongsan, 280 Daehak-Ro, South Korea. 2 Professor, Department of Computer Engineering, Yeungnam University, Gyeongsan, 280 Daehak-Ro, South Korea. Abstract This paper proposes to analyze the concentration of learning. It is a system to detect and analyze features of facial expressions in the video data for the verification of the leaner s concentration. The leaner s concentration is important to class effectiveness for learners. The facial expression changes depending on the leaner s unusual environment and emotion. This system analyzes and reflects the learner s current situation. First, we detect the learner s face on the learning video image and detect feature points in the detected face. Second, we decide the learner s facial criteria value and analyze coordinate values of feature points. We decide criteria values of the frontal face, opened eye and closed mouth. Third, we compare values of detected feature points with criteria values and decide the turned face, the downed face, the closed eye, the opened mouth and facial emotions (smile, surprise, sadness, anger). Lastly, we determine the state of concentration and the state of non-concentration while learning. It is not concentrated when coordinates have out of criteria values and it is concentrated when coordinates have criteria values. We have confirmed the learner's concentration by the changed amount of the distribution of feature points using the proposed system. Keywords: Analysis of the concentration, learner's concentration, feature point, analysis of the image, movement of face, analysis of facial emotion, facial expression. INTRODUCTION In recent years, research in the field of facial recognition has rapidly developed. Facial recognition has been applied to various fields such as facial expression recognition and iris recognition. There are other types of recognition such as facial recognition, pedestrian recognition, as well as movement recognition, such as hands in human recognition. Additionally a number of other types of recognition, not specifically related to human recognition, exist and include items such as, vehicle recognition, license plate recognition, and several object recognitions. In particular, the research of facial recognition continues in the facial image and security field. The research related to the facial image on the learning lacks in the field of education. In this paper, we propose a method to analyze concentration of learning through the detection of the facial feature points in the learner's facial expression image. We convert it to coordinate after taking a variety of facial images during the learning process. In order to determine whether the learner s state is concentration or non-concentration, we examined the status of a smile, surprise, sadness and anger on facial images. This paper is organized as follows. First, we described the concentration of the learning by related works and introduce algorithms of the face detection and feature extraction. Second, we described in detail about system design to analyze the learner's concentration using detection system of facial feature points proposed in this paper. Finally, we described conclusions and described limitations and future researches about this study. RELATED WORK The learner's concentration is important [1][2][3] to class effectiveness for learning. The opposite of concentration is non-concentration, distraction and dispersion. Research studies related to concentration are divided into two areas. There are researches for improving and measuring the concentration. The study to measure the learner s concentration on the learning environment of the e-learning suggests the ways to measure the concentration while learning after examining the learner s eye and brain waves [4]. When a learner in an e-learning context bows his head to write, the study has the way that delays the video speed to examine the learner s concentration state at the same time [5]. We use the Viola-Jones [6] algorithm, which is one of the appearancebased methods for face detection. We extract feature points by using the Shi & Tomasi [7] algorithm. DESIGN OF THE LEARNER S CONCENTRATION ANALYSIS SYSTEM We examine changes of the learner's facial features in the video image. In learner's facial features, we check the following facial features the smile, surprise, sadness, anger, closed mouth, opened mouth (the shape of the mouth is changed based on pronunciation), opened eye, closed eye, frontal face, turned face and downed face

2 In order to find the feature points of the face, we extract facial feature points after detecting the facial area. We set default values of the face, the eye and the mouth in the coordinate of extracted feature points and we determine the concentration state and the non-concentration state by comparing default values and various values of learner's facial features. The concentration is defined as the situation of the opened eye, the closed mouth and the frontal face in the situation of concentration on the learning. The non-concentration is defined as the situation of the closed eye, the opened mouth, the downed face, the turned face and facial expressions of emotion in the situation of nonconcentration while learning. If the length value of the eyes is less than the criteria value and the value occurs above 0.9ms, we determine the state of the closed eye. It determines the state of non-concentration while learning. If that value is larger than the criteria value and the value occurs under 0.9ms, it is the blinking eye. The blinking eye is the state of concentration while learning. Figure 1. Criteria values First, we detect the face in the learner s image and extract facial feature points. We determine criteria values of the frontal state. These values determine with the coordinate values of the face, the eyes and the mouth of the frontal facial feature point. Next, after determining criteria value, we analyze by comparing the learner's state and recognize facial expressions. The determination of criteria values and recognition of facial expressions are explained in details. Lastly, we determine whether the learner's state is concentration or nonconcentration. A. Definition of the concentration and non-concentration In this paper, the criteria value of concentration or nonconcentration is defined as follows: B. Criteria value Fig. 1(a) is the frontal facial length, Fig. 1(b) is the facial center, Fig. 1(c) is the length of the opened eye, Fig. 1(d) is the length of the closed mouth, and Fig. 1(e) is the width of the closed mouth. With these, we determine five criteria values in the state of the frontal face. After the determination of criteria values, we compare criteria values with the learner s coordinate values of learning, and then we determine the state of the turned face, the downed face, the closed eye, facial expressions of emotions and the opened mouth. After we compare criteria values (see Fig. 1) with extracted feature points, we determine the downed face, turned face, closed eye, opened mouth, and the expressions of facial emotion (see Fig. 2). Fig. 1(a) is the minimum value of Y coordinate of the eye and the maximum value of Y coordinate of the mouth, this is determined as a criteria value of the facial length. And we detect the difference value of the maximum and minimum of the Y coordinate of the facial length from learner s coordinate value on the learning, if the value is less than criteria value we determine the state of the downed face (see Fig. 2(h))

3 Figure 2. The analysis of the feature points. (a): frontal face; (b): closed eye; (c): smile; (d): surprise; (e): sadness; (f): anger; (g): turned face; (h): downed face; (i): [ah]; (j): [ae]; (k): [e]; (l): [o] In order to determine the state of the turned face, we determine criteria values of the facial center (see Fig. 1(b)). It is the maximum value of the X coordinate of the right eye and the minimum value of the X coordinate of the left eye. We detect the different values of the maximum and minimum of the X coordinate of the eye with extracted feature points from learner s coordinate value while learning. If the value is less than criteria value, we determine it as the state of the turned face to the left (see Fig. 2(g)). If the value is larger than that, we determine it as the state of the turned face to the right. So, if that value is larger or less than the criteria value, both determine the state of the turned face. In order to compare the closed eye with the opened eye, we determine criteria value of the opened eye (see Fig. 1(c)), This value is the maximum value and the minimum value of the Y coordinate of the eye. Thus the different values of the maximum value and the minimum value of the Y coordinate of the eye are shown from learner s coordinate value while learning. If the value is less than criteria value and the value occurs over 0.9ms, we determine the state of the closed eye (see Fig. 2(b)). If that value is less than 0.9ms, that is the blinking eye. In order to decide the opened mouth, we determine criteria value of the closed mouth (see Figs. 5(d) and 5(e)). These are the maximum and the minimum values of the Y coordinate and X coordinate of the mouth. Thus different values of the maximum and the minimum values of the Y coordinate and X coordinate of the mouth from learner s coordinate value of learning, if the different value is larger than criteria value of the X coordinate, we determine the state of the opened mouth ([ah], [ae], [e]) (see Figs. 6(i), 6(j) and 6(k)). If the different values are less than the criteria values of the X coordinate, we determine the state of the opened mouth ([o]) (see Fig. 2(l)). If the different values are larger than the criteria values of the Y coordinate, we determine the state of the opened mouth ([ah], [ae], [e], [o]). Thus, if these values are less or larger than the criteria values, both determine the state of the opened mouth. We determine expressions of facial emotion (smile, surprise, sadness, anger) by using coordinate values of the mouth and the eyes. In order to decide the smile, surprise, sadness and anger, we determine criteria values of the opened eye and the closed mouth (see Figs. 5(c), 5(d) and 5(e)). Thus different values of the X and Y coordinate of the eye and the mouth from learner s coordinate value while learning. If different values of the X and Y coordinate of the mouth are larger than criteria value and different values of the Y coordinate of the eye are less than criteria value, we determine the state of the smiling face (see Fig. 2(c)). Thus, different values of the X and Y coordinate of the eyes and the mouth (see Fig. 2(d)) contain different values of the X and Y coordinate of the mouth and are more than the criteria values and difference values of the Y coordinate of the eye that is larger than criteria value, we determine the state of the surprised face (see Fig. 2(f)). Also if different values of the Y coordinate of the eye are less than criteria values, we determine the state of the sad face (see Fig. 2(e)). If the different values of the Y coordinate of the eye are larger than criteria values, we determine the state of an angry face (see Fig. 2(f)). Thus, if the difference value of the Y coordinate of the eye is less or larger than criteria value, both determine states of the facial expressions of emotions (smile, surprise, sadness, anger). EXPERIMENTS We take a video image on the learner s state and we detect the learner s face in the image. We determine criteria values after we extract coordinate values of feature points in the detected face. From this, we determine whether a student has the frontal face, the downed face, the turned face, opened eyes, closed 11346

4 eyes, expressions of emotion (smile, surprise, sadness, anger) or the opened mouth ([ah], [ae], [e], [o]) with the detected coordinate values, and then the state of the concentration while learning is determined. Fig. 3 observes the changed state of objects for three minutes which shows the changed feature points in the face image after taking the learner s video image. With this, we identify the state of the downed face, the turned face, closed eyes, smile, surprise, sadness, anger and the opened mouth by the changed distribution of feature point in each frame. If feature points get out of the criteria range (see in Fig. 3), those are states of the turned face, closed eyes, smile, surprise, sadness, anger and the opened mouth. The period where the facial center value fell sharply (see Fig. 3(a)) is the state that the object is actually turned his head to the left. The changed facial length (see Fig. 3(b)) is actually the nonconcentration state of the surprise, anger and opened mouth ([ah], [ae] and [o]) and if there is the period where the facial length value fell sharply, this period is actually the period of the downed face. This states the changed width of mouth (see Fig. 3(c)) is actually the non-concentration states of the smile, surprise, turned face and opened mouth ([ae], [e], [o]). The changed length of mouth (see Fig. 3(d)) is actually the non-concentration state of surprise, sadness, downed face and opened mouth ([ah], [ae], [o]). Red circles (see Fig. 3(e)) are states of the blinking eyes for 0.1 seconds. Blinking eyes appear six times for three minutes. These are states of blinking eye in the real situation, these are not state of the non-concentration. The others show the changeable length values of the eyes, these are states of nonconcentration of smile, surprise, sadness, anger and closed eye in real situations. Red arrows except red circles (see Fig. 3) were actually nonconcentration states of the closed eye, opened mouth, downed face, turned face and facial expressions of emotion (smile, surprise, sadness, anger), we observed the same state in the chart. The rest was actually the concentration state of the frontal face with the opened eye and closed mouth in which we also observed as the same state in the chart

5 Figure 3. The analysis of the learner s state (a): Turned face; (b): Length of face; (c): Width of mouth; (d): Length of Mouth; (e): Closed eye As the Result of the experiments on 30 students in a classroom, the detection rate of feature points and the comparison value of each of the detected features are shown in Table 1. The detection rate of smile, surprise, sadness, anger, turned face, downed face, closed eye, opened mouth ([ah], [ae], [e], [o]) are 95.31%, 93.94%, 89.98%, 88.76%, 94.45%, 95.24%, 93.94%, 92.63%, 90.12%, 87.25% and 97.45% respectively. The detection of the smile, turned face, opened mouth ([ah], [o]) has a high detection rate, but the detection rate of the sadness, anger and opened mouth ([e]) is lower than the others. Problems occurred in the experiment are as follows. When the object is suddenly far away relatively, the detection rate is lower. Also, if more light shines on one side of the face, the detection rate is lower. Table 1: Detection Rate and Analysis of Detected Features. X: x coordinate value; Y: y coordinate value Facial Movement Detected Feature Values & Criteria value Detection Rate ( % ) Eye Mouth Facial length Facial center y x y y x Smile Small Large Surprise Large Small Large Large Sadness S Small Anger Large Large Large

6 Turned Face Small Small or Large Downed Face Small Small Closed Eye Small [ah] Large Large [ae] Large Large Large [e] Large [o] Small Large Large CONCLUSION In this paper, we detected facial feature points for concentration analysis of learners. We determined the state of the non-concentration from out of criteria values of facial feature points. We confirmed whether the learner is the state of concentration or non-concentration in relation to how the learner s face falls down or turns from side to side, or if their eyes open or close or their mouth moves to the up and down or moves to the left and right based on the analysis of data. ACKNOWLEDGEMENT The corresponding author is Wookhyun Kim. [5] Joohee Kang, Minjea Park, Yujung Yun, Hoyang Choi, Seongwon Park and Kwangsu Cho, Study on e-learner's Attention improvements, Korea HCI society conference, pp , Feb [6] P. Viola and M. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp , [7] Shi, J., and C. Tomasi, Good Features to Track, IEEE Conference on Computer Vision and Pattern Recognition, pp , Jun REFERENCES [1] Seunghui Cha, Wookhyun Kim, An Analysis Method of Class Concentration by Facial Feature Detection, international Information Institute (Tokyo). Information vol.18. no.6(b) p , Jun [2] Seunghui Cha, Wookhyun Kim, Concentration analysis by detecting face features of learners, Communications, Computers and Signal Processing (PACRIM), 2015 IEEE Pacific Rim Conference on, p.46-51, [3] Seunghui Cha, Jong Wook Kwak, Wookhyun Kim, Performance Analysis of Face Detection Algorithms for Efficient Comparison of Prediction Time and Accuracy, Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), p , [4] Hyung-Mo Ahn, Sang-Cheon Nam and Ki-Sang Song, Application of bio-signal measurement to identify learning concentration in e-learning environment, The Korean association of computer education, Vol. 16, pp ,