Facial Event Classification with Task Oriented Dynamic Bayesian Network

Transcription

1 Facial Event Classification with Task Oriented Dynamic Bayesian Network Haisong Gu Dept. of Computer Science University of Nevada Reno Qiang Ji Dept. of ECSE Rensselaer Polytechnic Institute Abstract Facial events include all activities of face and facial features in spatial or temporal space, such as facial expressions, face gesture, gaze and furrow happening, etc. Developing an automated system for facial event classification is always a challenging task due to the richness, ambiguity and dynamic nature of facial expressions. This paper presents an efficient approach to real-world facial event classification. By integrating Dynamic Bayesian Network (DBN) with a general-purpose facial behavior description language, a task-oriented stochastic and temporal framework is constructed to systematically represent and classify facial events of interest. Based on the task oriented DBN, we can spatially and temporally incorporate results from previous times and prior knowledge of the application domain. With the top-down inference, the system can make active selection among multiple visual channels to identify the most effective sensory channels to use. With the bottom-up inference from observed evidences, the current facial event can be classified with a desired confident level via the belief propagation. We applied the task-oriented DBN framework to monitoring driver vigilance. Experimental results demonstrate the feasibility and efficiency of our approach. 1. Introduction The general-purpose facial expression analysis has been explored for decades. Numerous techniques have been proposed. A recent survey of existing works can be found in [1]. In terms of accuracy and robustness, it is still hard for real-world applications. The difficulty comes from two folds. One is the richness and complexity of facial expressions. The number of expressions occurring daily is more than It is hard to accurately classify them, even by an experienced human expression coder. The other is the variety of expression appearances. Expressions can appear in the form of geometrical deformation of feature points, eye movements,or furrow happenings. A single visual cue will not be able to efficiently capture all of these changes. However, each specific application only involves several limited expressions of interest. A task-oriented approach can be used to remove the ambiguity due to rich expressions. Furthermore, the multiple visual cues provide several effective sensing channels. By actively selecting the most informative channel, the effective features can be captured so as to make the recognition highly efficient. These two methods constitute the fundamental parts in our facial event classification system. In order to infer mental states from measured facial information, one quantitative description system for facial expression is necessary. Ekman s group [2] proposed an unified description method of expression: Facial Action Coding System (FACS). It totally has 71 primitive units, called Action Unit(AU). Based on them, any expression display can be represented by a single AU or the AU s combination. So far some research groups have explored the FACSbased classification [3, 4, 5, 6]. FACS representation of facial expression, however, is static and deterministic. Facial event develops over time and the detected facial features contain uncertainties. Recognizing facial events solely based on FACS is inadequate. In this paper, we introduce a dynamic and stochastic facial expressions representation framework based on the Dynamic Bayesian Networks so as to efficiently realize the recognition of facial events from an image sequence. Figure 1: the DBN based recognition Fig. 1 shows our DBN-based processing flow. For each application, a task-oriented DBN is created with the expression database of the application domain. Based on FACS, each facial event of interest is disassembled up to single AUs through several layers. Each single AU associates with its most informative sensor(s). With the Bayesian network,

2 we not only classify what is the current facial event, but also determine which sensing channel is the most effective to use next. 2. Multiple visual channels The face is a rich resource to infer the mental state. Different mental states often appear in different forms of facial features. So a sensing system providing with multiple visual cues is required to efficiently extract the information from faces. So far, we have developed different methods for facial sensing, which include: 1. IR based eye detector, which uses IR camera to robustly detect and track the eye pupils in real-time [7]. 2. Facial feature tracking system. Each landmark on face is modeled with a set of Gabor wavelets. Simultaneously using the global head movement and the smoothing constrains realizes a practical tracking system for multiple facial features under real-world variable conditions [8]. 3. furrow detection. As the result of facial feature tracking, each tracked facial landmark is identified with its physical meaning. Based on the location of feature points, the region is set for each possible furrow. The presence of furrows and wrinkles within the targeted regions is determined by edge feature analysis. 4. Head orientation estimation, which utilizes the correlation between 3D head orientation and properties of pupils. We build a so-called Pupil Feature Space(PFS) which is constructed by seven pupil features: such as, inter-pupil distance, pupil size, orientation, etc. The PCA method is used to represent pose distribution in PSF space. The head orientation can be determined by mapping the pupil parameters to PFS space. 5. Head motion estimation. The results of facial feature tracking provide the spatial relationship among feature points in forms of local facial graph. Based on the 2D local similarity transformation, we can determine the inter-frame global similarity parameters within the in-plane motion. Then, by using the tip of nose as the center we further map them to the feature points in the current frame by a scaling transformation. The obtained zooming parameter can be used to determine the head motion in depth. All of the above methods are implemented in nearly realtime mode. The detailed description of these methods can be found in[7, 8, 9] driver vigilance, we try to classify the drowsy related facial events to identify in-vehicle driver states. The events related to Inattention, Yawning and Falling asleep are targeted. The single AUs associated with these fatigue events are shown in Fig. 2. Their corresponding descriptions are presented in Table 1. AU7 AU9 AU25 AU26 AU27 AU43 AU51/52 AU55 AU56 AU58 AU61 AU62 Figure 2: Action units for components of fatigue facial expressions (some adapted from [2]) Facial expression is the entire facial behavior. However, the single AU usually focuses on local feature movement. In this paper, we propose a whole-to-part model structure to connect the single AUs and its entire facial event. Fig. 3 shows the spatial relationship of entities related to each vigilance level with local facial features and the associated visual channels. It mainly consists of 3 layers: Entire facial display, Partial facial display and Single AU. The entire display layer includes all of the events (or vigilance levels) of interest. In the partial display layer, we divide an entire facial display into partial display of local facial features, such as eye motion, face gesture, upper face and lower face. The connection between the entire and partial layers is based on each entire display and its corresponding feature appearances. In the single AU layer, all of related single AUs are listed and connected to its partial facial display according to FACS. Table 1: Descriptions of AUs in Fig. 2 AU# Descriptions AU# Descriptions AU7 Lid Tightener AU9 Nose Wrinkle AU25 Lips part AU26 Jaw Drop AU27 Mouth Stretch AU43 Eye Closure AU51 Head Turn Left AU52 Head Turn Right AU53 Head Up AU55 Head Tilt Left AU56 Head Tilt Right AU58 Head Back AU61 Eyes Turn Left AU62 Eyes Turn Right 3 Facial event classification 3.1. Spatial Dependency A particular application always focuses on a limited number of facial events of interest. For instance, to monitor Figure 3: BN for vigilance detection.

3 Since in this spatial modeling, the nodes within each layer are conditionally independent and are non-exclusive with each other, and the connection between the different layer is casual, we can build a Bayesian Network (BN) for facial event analysis. In the BN model, the top layer can be viewed as a simple Naive Bayes classifier. It consists of hypothesis variable C with three mutually exclusive states c 1,c 2,c 3, for inattention, yawning and falling asleep respectively. The hypothesis variable serves as a parent of three children nodes A 1, A 2, and A 3 corresponding to the facial events Ia,Y w and Fa respectively. The goal for this layer is to find the probability of state c i given A j = a j : Pr(C = c i A 1 = a 1,A 2 = a 2,A 3 = a 3 ) Parent Nodes Table 2: Probabilities Child Nodes Fatigue Ia Yw Fa Level T F T F T F Inattention Yawning Falling asleep In other words, this probability represents the chance of the class state c i when each attribute variable has the value where A j = a j. When the probability of specific state is maximal, it has the largest chance in which the facial event belongs to the variable c i. The conditional probabilities between the states of the hypothesis variable and the corresponding event variables are given in Table 2. Since in our case A i is unobservable, values of A i should be inferred through its lower layers. H K E I G F C Z F J Y O N D A B J F F X I E Figure 4: The geometric relations of facial features and furrows In the intermediate layers, the spatial relationships are represented by one of the typical BN connections: Serial, Diverging, and Converging connection[10]. In the bottom layer, each single AU associates with specific movement measurement of facial features. We connect each single AU with its effective visual channel. Table 3 gives the quantitative description of each AU, its quantification using the feature marked in Fig. 4 and the associated visual channels Temporal dependency A natural facial event evolves over time from the onset, the apex and the offset. The facial appearance can be classified to the underlying mental state by accumulating evidences C G K H l1 up to the apex. Static BN (SBN) modeling of facial expression works only with visual evidences and beliefs from a single time instant. It lacks the ability to express temporal relationship and dependencies in a video sequence. In order to overcome this limitation, we use Dynamic BNs (DBN) for modeling the dynamic aspect of the facial event. Table 3: Measurement of AUs AU Method Appearance Sensing Channel AU7 IFH non-increased Upper face Facial tracker and HGF increased AU9 wrinkle increased Upper face Furrow detector in JFF J AU25 DB < T 1 Lower face Facial tracker and NA non-increased AU26 T 1 < DB < T 2 Lower face Facial tracker AU27 DB > T 2 and Lower face Facial tracker l1 increased AU43 Pupils lost for a while pupil Eye tracker AU51 OZ turn left Head Gaze detector Orientation AU52 OZ turn right Head Gaze detector Orientation AU55 Head moves Head motion Head motion left-down detector AU56 Head moves Head motion Head motion right-down detector AU58 Head moves back Head motion Head motion detector AU61 E and E move left pupil Eye tracker AU62 E and E move right pupil Eye tracker * Note: T 1 and T 2 are predefined thresholds. In Fig.5, the DBN is made up of interconnected time slices of SBNs as exactly described in the preceding section. The relationships between two neighboring time slices are modeled by the first order Hidden Markov Model, i.e., random variables at temporal instance T are affected by observable variables at T, as well as by the corresponding random variables at preceding temporal instance T 1 only. The evolution of facial expression is defined by moving one time frame in accordance with the process of video sequences, so that the visual information at the previous time provides classification support for current hypothesis. Eventually, the belief of the current hypothesis of the mental state is inferred relying on the combined information of current visual cues through causal dependencies in the current time slice, as well as the preceding evidences through temporal dependencies. Figure 5: DBN in terms of temporal instance T In contrast to one frame as the temporal instance used in pervious DBN based sequence analysis, we also define another time scale in the task oriented network, that is the phase instance. One phase instance is a continuous facial

4 change which proceeds for a limited period of time covering a single ONSET, APEX, or OFFSET duration. The temporal dependency in the complicated facial event can be modeled by the phase instance as well as the frame instance. At each frame instance, the bottom-up inference is executed to classify each current facial display. In each phase instance, the top-down inference is conducted to actively select effective visual channel(s). In the next subsection, we will give the details on the facial event inference mechanism for Two scale active inference mechanism Figure 6: DBN in terms of phase instance We cast the multiple visual channel system into a DBN framework, while the DBN provides a dynamic knowledge representation and control structure that allows sensory information to be selected and combined according to the rules of probability theory and current recognition tasks. Usually in order to find the best feature region, we need to exhaustively search all the visual channels. However, the task-oriented approach provides an efficient method of purposive selection. As described before, in the temporal dependency of our DBN, there exits two time scales. One is the phase instance which usually consists of several consecutive frames, the other the frame instance. Fig. 6 depicts an example of temporal dependency in term of phase instances. Each facial event recognition will proceed two phases : Detection and Verification. At first, the Inattention detection (Ia-Detection) phase is triggered. With the specified detection target, in this phase the related visual channels are selected and activated, based on the top-down inference of our task-oriented BN. After one Inattention facial display is detected, the system evolves into the Inattention verification (Ia-Verification) phase. Meanwhile the Yawning detection (Ya-Detection) phase is triggered. In this way, we can determine the entire targeted facial displays according to the current phase instance. From the entire facial display, a top-down BN inference is conducted to identify the associated single AUs so as to activate the corresponded visual channels. Once the phase instance changes, we conduct the decision-making for visual channels and also adjust the parameters of CPTs in the same BN structure according to the current phase configuration. From each frame slice, the system checks the extracted information from activated visual channels and collects the evidences. With the observed evidences, a bottom-up BN inference is conducted to classify the current facial display. We summarize the purposive sensing in one recognition cycle as follows: 1. Set entire facial displays in the detection phase. With the top-down BN inference, identify the associated single AUs and activate the corresponding visual channels. 2. In each frame slice, classify the current facial display by the bottom-up BN inference from the observed evidences and seek the ONSET of the facial event by the curve of the posterior probability. 3. If the ONSET is identified with a high confidence, evolve into the verification phase. Otherwise goto step Based on the targeted display(s) in the verification phase, the top-down inference is used to activate the most informative visual channels From the activated visual channels, collect the observation evidences and seek the APEX of the current facial event frame by frame. 6. If the APEX in the verification phase is identified with a high confidence, reset the recognition cycle, goto step 1. Otherwise goto step 5. 4 Experimental Results 4.1. Static facial event classification Figure 7: Some typical facial displays in fatigue sequences with tracked features attached Fig. 7 collects some frames of typical expressions from two different fatigue sequences. Fig. 7(a), (b), (c) depict the neutral state, inattention and yawning state of one person. Fig. 7(d), (e), (f) show the inattention, yawning and falling asleep states in the other sequence. In the fatigue detection oriented Bayesian Network (Fig. 3), we set the prior probabilities for all the levels (inattention,yawning, and falling asleep) to be the same to conduct the bottom-up classification from the observed evidences. Table 4: Static classification No. Observation Evidences Results ED OE HD FT FD (Ia) (Yw) (Fa) (a) x x x x x (b) x AU51 x x x (c) x x x AU7 AU AU27 (d) x AU52 x x x (e) x x x AU7 x AU27 (f) x AU51 AU55 x x Table 4 shows observed evidences from each visual channel and the classification result for each image. ED, 1 The most informative channel is defined as the one that can maximally reduce the uncertainty of the hypothesis

5 OE, HD, FT and FD in the table indicate Eye Detector, Head Orientation estimator, Head motion Detector, Facial Feature Tracker and Furrow Detector, respectively. (Ia), (Yw) and (Fa) represent Inattention, Yawning and Falling asleep, respectively. x means no evidence. The (a) row of Table 4 shows that there is no fatigue-related evidence available from all the sensory channels in the image of Fig. 7(a). The posterior probability of each fatigue state obtaining from the Static BN inference has the same value as It means the person is in the neutral state, or the states in which there are no fatigue related facial displays. The row (b) in Table 4 shows that in Fig. 7(b) the evidence of AU51 is detected from the visual channel(oe). The maximum posterior probability is obtained in the Inattention. The facial event is classified as Inattention. The row (c) shows that AU7, AU27 and AU9 are detected from the visual channels FT and FD, respectively. The classification result is the Yawning. The row (d) and (e) show that the corresponding classification results are Inattention and Yawningstates, respectively. The row (f) shows that there are AU51 and AU55 detected from the channels OE and HD, respectively. The posterior probabilities of Inattention and Falling asleep are the same(0.47). At this moment, there exists ambiguity in the static facial display. The system is difficult to identify fatigue states. The above static recognition experiments verify that a good classification can easily be obtained for typical facial displays with the task-oriented BN. However, when the ambiguity exists among the displays, or in the transition period of states, the additional temporal information is often required to resolve the ambiguity Dynamic event classification A moderate fatigue sequence From the frame 21 to 185, the posterior probability of Inattention keeps the highest among the three states. In this period, the probability curve of Inattention goes down several times because of neutral states between different Inattention facial displays. At these moments, there are no fatigue related evidences observed at all. With the temporal dependency from previous classification results, the mental state of the subject is still classified as Inattention although the value is comparatively lower. After the frame 185, the Yawning evidences (AU25, 26, 27) are consistently detected. The probability curve of Yawning increases gradually. During this period, the yawning curve also reduces one time because evidences of both Yawning and Inattention are detected simultaneously at around the frame 200. Figure 9: The posterior probability curves The experiments of the sequence validates that the DBNbased approach successfully integrates temporal information in previous moments to remove the ambiguity due to multiple conflict evidences or transition periods Dynamic inference and sensing Figure 8: One video clip with a blended facial displays of neutral, Inattention and Yawning state Fig. 8 shows a moderate fatigue sequence, which consists of samples from one 210-frame sequence at the interval of 10 frames. The clip starts from 20-frame neutral states. Then several facial events of Inattention are preformed with some neutral states in the transition moments. Finally the subject opened his mouth and preformed facial events of Yawning. Fig. 9 depicts the posterior probability curves of the three fatigue levels, obtained by the task-oriented DBN system. We uniformly set the transitional probability between two consecutive temporal slices as 0.5. In the first 20 frames, there are no fatigue related evidence extracted. The probability values of three levels are the same as Figure 10: a typical fatigue sequence Fig. 10 shows samples from a 120 frame sequence at the interval of 5 frames, which consists of Inattention, Yawning and Falling asleep states in order. Falling Asleep Yawning Inattention Image Frame Figure 11: The posterior probability curves Fig. 11 depicts the posterior probability values of three states obtained from the inference of DBN. We can see

6 in this graph that the confident level goes down due to some supporting evidences missing, such as at the frames 70,80. Because of the integration of the previous results and the knowledge of application domain, the classification is still stable even at these moments. The frame 110 here corresponds to Fig. 7(f) in the static experiment. The obtained posterior probabilities of Inattention, Yawning and Fallingasleep are ,0.0034, and , respective. The temporal dependency based on DBN significantly removed the ambiguity in the static classification, shown as in the row (f) of Table 4. Figure 12: Sensor selection Fig. 12 depicts the selection of activated visual channels during the recognition of the above clip. According to the task-oriented DBN, the Inattention related visual channels, which are Head motion Detector (HD), Head Orientation estimator (OE) and Eye Detector (ED), were initially activated. At frame 5 the evidence of Inattention was detected from the OE channel. The system started to focus on OE. At frame 20, the ONSET of the Inattention display was detected based on the curve of the posterior probability of Inattention in Fig. 11. So the system finished the detection phase of Inattention and evolved into the verification phase of Inattention. At the same time the detection phase of Yawning was triggered and the associated visual channels, which are Facial Feature Tracker(FT) and Furrow Detector(FD) were activated. At frame 40, the system identified the APEX of the Inattention expression and verified the Inattention state. After that, at frame 45 the new evidence on Yawning was detected from the FT channel. The system focused on it. At frame 60, the ONSET of the Yawning expression was identified. The system evolved into the verification phase of Yawning and the detection phase of Falling asleep. The HD channel was activated. At frame 75 and 85, one APEX of Yawning was identified and the Yawning state was verified, respectively. From frame 85, the new evidence on Falling asleep was detected and the system focused on the HD channel. At frame 95, the ONSET of the Falling asleep was identified. The system further evolved into the verification phase of the Falling asleep. Finally the APEX of the Falling asleep was identified by the curve of the posterior probability of Falling asleep in Fig. 11. The experimental sequences here simulate typical fatigue evolution processes, with only a short transition period. In reality, the Yawning states sometimes can last a certain long period and don t evolve into the Falling asleep state. In this case, the activated channels will keep the same for a while. Sometimes, the Yawning expression may disappear for some reasons. In this case, the system cannot detect any evidences of Yawning for a certain period time. All three posterior probabilities tend to lower and will be the same after a while. The fatigue states will disappear at this point. The system will reset to the detection phase for Inattention. 5 Conclusion In this paper, we presented a practicable and efficient framework for real-world facial event recognition. The proposed method has several favorable properties: A dynamic and stochastic facial expressions representation framework based on combining FACS and DBNs is proposed. This framework can be adapted for different applications. The domain knowledge and previous analysis results are successfully integrated systematically to remove ambiguities in facial event displays. The selective sensing structure among the multiple visual channels makes the recognition of facial events very efficiently and can easily extend to encapsulate other new effective visual channels, such as estimation of gaze and eyelid movement, even for sensors of different modalities. This project is supported in part by a grant from AFOSR under the grant number F References [1] M. Pantic and L. Rothkrantz, Automatic analysis of facial expressions: The state of the art, IEEE Trans. Pattern Anal. Machine Intell., vol. 22, no. 12, pp , [2] P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System (FACS): Manual. San Francisco, CA: CD Rom, [3] J. Zhao and G. Kearney, Classifying facial emotions by backpropagation neural networks with fuzzy inputs, in Proc. Int l Conf. Neural Information Processing, pp , [4] M. Pantic and L. Rothkrantz, Expert system for automatic analysis of facial expression, J. Image and Vision Computing,vol.18, no.11, pp , [5] Y. Tian, T. Kanade, and J. F. Cohn, Recognizing action units for facial expression analysis, IEEE Trans. Pattern Anal. Machine Intell., vol. 23, no. 2, pp , [6] J. J. Lien, T. Kanade, J. F. Cohn, and C. Li, Detection, tracking, and classification of action units in facial expression, Int l J. Robotics and Autonomous System, vol. 31, no. -, pp , [7] Z. Zhu, Q. Ji, K. Fujimura, and K. Lee, Combining Kalman filtering and mean shift for real time eye tracking under active ir illumination, in Proc. Int l Conf. Pattern Recognition, Aug [8] H. Gu, Q. Ji, and Z. Zhu, Active facial tracking for fatigue detection, in IEEE Workshop on Applications of Computer Vision, (Florida, USA), [9] Q. Ji and Z. Zhu, Eye and gaze tracking for interactive graphic display, in Smart Graphics, (Hawthorne, NY, USA), [10] F. V. Jensen, Bayesian Networks and Decision Graphs. Springer- Verlag: Springer-Verlag New York Inc., 2001.