Filipino Sign Language Recognition using Manifold Learning

Size: px

Start display at page:

Download "Filipino Sign Language Recognition using Manifold Learning"

Elvin Butler
6 years ago
Views:

1 Filipino Sign Language Recognition using Manifold Learning Ed Peter Cabalfin Computer Vision & Machine Intelligence Group Department of Computer Science College of Engineering University of the Philippines Diliman Rowena Cristina L. Guevara Digital Signal Processing Laboratory Electrical and Electronics Engineering Institute College of Engineering University of the Philippines Diliman Prospero C. Naval, Jr. Computer Vision & Machine Intelligence Group Department of Computer Science College of Engineering University of the Philippines Diliman ABSTRACT Sign Language is at the core of a progressive view of deafness as a culture and of deaf people as a cultural and linguistic minority. An in-depth study of Filipino Sign Language (FSL) is crucial in understanding the Deaf communities and the social issues surrounding them. Computer-aided recognition of sign language can help bridge the gap between signers and non-signers. In this paper, we propose Isomap manifold learning for the automatic recognition of FSL signs. Video of isolated signs are converted into manifolds and compiled into a library of known FSL signs. Dynamic Time Warping (DTW) is then used to match the nearest library manifold with the query manifold for an unknown FSL sign. 1. INTRODUCTION The World Health Organization (WHO) defines hearing impairment as total or partial loss of hearing on one or both ears. The levels of impairment could be mild, moderate, severe or profound. WHO defines deafness as the complete loss of ability to hear from one or both ears [17]. The 2000 Census on disability [13] reported 121,000 Filipinos with total or partial hearing loss. This is a fraction of an estimated one million Filipinos with Disabilities in the Philippines. A primary binding force for the Filipino Deaf is the language they use Filipino Sign Language (FSL). Sign language is at the core of the progressive view of deafness as a culture, and of deaf people as a cultural and linguistic minority [1]. Over half of the Deaf respondents in a study done by the National Sign Language Committee declared Filipino Sign Language as their mode of communication [3]. Unfortunately, many people do not know that there is a natural sign language used by the Deaf communities ([4],[2]). Interpreting organizations or programs in 15 regions in the Philippines are very difficult to find and there is an unequal distribution of education programs that use sign language [3]. Additional challenges come from lack of documentation about regional variations of the signs ([14],[15],[11],[2]). FSL is a key component in understanding the Deaf communities and the social issues surrounding them. Automatic analysis of FSL will make linguistic research easier, and computer-aided interpretation will help bridge the gap between signers and non-signers. It is hoped that this research can contribute to these goals. 2. FILIPINO SIGN LANGUAGE Sign Language is the natural language of the Deaf. Users of sign language are called signers. Sign language is a visual language and signers use their hands, arms, shoulders, torso, neck and face to communicate [12]. One misconception about sign language is there is only one universal, international sign language. This is incorrect since there are at least a hundred recognized sign languages in the world [8]. This study will focus on the sign language used by the Deaf communities in the Philippines. Much like spoken languages, numerous variations of FSL have been observed in the field. To reduce the scope of work, only traditional signs were used and native signers from Metro Manila were considered. Traditional signs are defined as signs used by a large part of the communities and has been around for decades. In contrast, emerging signs are defined as signs that have come into use only in the last five years or so [14]. It is only recently that documentation of indigenous signs and their origins have started [14],[2]. In spoken languages, the basic unit of utterance is called a phoneme. Similarly, the basic unit in sign language is also called a phoneme, even though they are not based on sound [7]. Although signs are often decomposed into five parameters (see 2.2), there is no consensus yet on sign language phonemes [16]. In addition, during conversations, non-sign gestures may be mixed in freely with signs. Facial expressions and body posture also play a large role in conversations as well [16], [12]. Signs are labeled by words called a gloss. This is a word borrowed from a written or spoken language to designate a particular sign; it is a linguistic tool and while the word

used is often the closest meaning of the sign, it is not a direct translation. When a phrase is used as a gloss, hyphens are inserted in place of spaces.

1 The Signing Space Signing space is a three-dimensional space from about the mid-torso to just above the head, extending forward from the chest to about one-arm length away, and extending about half

Movement can be grouped into two general categories: gross arm movement (tracking the path of the arms) and internal movement (changes in hand shape).

Liddel and Johnson further grouped these parameters into segments; one or more parameters occurring together form one segment.

2 used is often the closest meaning of the sign, it is not a direct translation. When a phrase is used as a gloss, hyphens are inserted in place of spaces. In sign linguistics literature, the gloss is often capitalized to distinguish it from regular use of the word [14], [11]. This paper shall follow that convention. 2.1 The Signing Space Signing space is a three-dimensional space from about the mid-torso to just above the head, extending forward from the chest to about one-arm length away, and extending about half an arm s length on both sides. It has been established in previous sign language research that during most signs, the hands and arms do not go beyond this space [14]. Movement can be grouped into two general categories: gross arm movement (tracking the path of the arms) and internal movement (changes in hand shape). Initial inventory of Filipino Sign Language observed over ninety hand shapes, approximately twenty hand locations, and six palm orientations [14]. Liddel and Johnson further grouped these parameters into segments; one or more parameters occurring together form one segment. Movement segments (M) are portions of the sign where the hands and arms are in motion; or when the hand shape is in transition. Hold segments (H) are portions of the sign where there is no motion or where the hand shape is in steady state. Signs are then composed of one or more segments [14]. For example, HMH means there is a Hold segment followed by a Movement segment followed by a Hold segment. Segment forms observed in FSL include H, M, MH, HMH, and MHMH [14]. (a) front view (b) quarter view Figure 1: Signing Space One or both hands may be used in signing, depending on the sign and the sign language. In the case where two-hands are used where only one hand is moving, the moving hand is called the dominant hand (DH) and the stationary hand is called the non-dominant hand (NDH) or the passive hand. Two-handed signs where both hands move in the same path and use the same hand shapes are sometimes called symmetrical signs. There are no left-handed or right-handed signs; one-handed signs may be performed with either left hand or right hand. Either hand may be used as the dominant hand in twohanded signs. In practice, right-handed people usually use their right hand for one-handed signs, finger-spelling, and as the DH in two-handed signs; left-handed people usually use the reverse. 2.2 Internal Structure of Sign Language Liddel and Johnson model sign language with five parameters [14]: 1. hand shape (or HS ) - described by which fingers and/or thumb are selected, extended or flexed 2. palm orientation (or just orientation ) - described by where the palm is facing 3. hand location (or just location ) - described by where the hand position is relative to the face, head, shoulders, arm and torso 4. movement - described by motion of fingers, thumb, hands and arms 5. non-manual signals (NMS) - which includes facial expressions and body posture (a) GIVE-HARD-WORK (b) COOK Figure 2: Examples of signs showing facial expressions and body posture. 3. MANIFOLD LEARNING 3.1 Dimensionality Reduction Many applications in computer science deal with complex data sets with many factors, variables or features. Analysis of data with many dimensions (features) is difficult and, past a certain point, algorithms fail to work. Reducing the number of dimensions is often done to simplify analysis and to reduce computational effort. Of course, we would like to preserve the underlying patterns and interaction of the variables as much as possible. The goal of dimensionality reduction then is to find a good approximation of the data with fewer dimensions. Principal Components Analysis (PCA) is one popular algorithm. Unfortunately, a major limitation of PCA is the requirement that the data lie on a linear subspace. Manifolds do not have this limitation [19], [18]. 3.2 Manifolds and Manifold Learning Manifolds are high dimensional mathematical structures that can be approximated by low dimensional shapes. To illustrate, let us take the globe as an example. While a globe is a three-dimensional object (a sphere), maps are two dimensional (planes). We can approximate a three dimensional shape using two-dimensional shapes. The goal of manifold learning then, given a data set described by many variables (high dimensions), is to look for a smaller set of variables (low dimensions) that can approximate the original data set.

Figure 3: (A) geodesic distance in blue, (B) shortest path in red, (C) Isomap of the data One well known manifold learning algorithm is Isomap [18]. 3.3 Isomap The Isomap algorithm extracts the embedded lower dimensional subspaces by extending classical Multi dimensional Scaling (MDS).

First a neighborhood graph is constructed, with each data point connected every other data point by edges with weights equal to the Eucledian distance between the data points.

Second, the shortest path between points are computed. Essentially, the geodesic distance is approximated by the shortest path distance.

3 Figure 3: (A) geodesic distance in blue, (B) shortest path in red, (C) Isomap of the data One well known manifold learning algorithm is Isomap [18]. 3.3 Isomap The Isomap algorithm extracts the embedded lower dimensional subspaces by extending classical Multi dimensional Scaling (MDS). Fig 3 will help illustrate the algorithm. The algorithm can be summariezd as follows: 1. First a neighborhood graph is constructed, with each data point connected every other data point by edges with weights equal to the Eucledian distance between the data points. Edges between points over a threshold are removed. Each point is connected only to its nearest neighbors. The threshold is either a maximum distance or k-nearest neighbors is used. 2. Second, the shortest path between points are computed. Essentially, the geodesic distance is approximated by the shortest path distance. Floyd s algorithm and Dijkstra s algorithm have been both used to find the shortest path distance, depending on the application [18]. Figure 4: PANGIT sign and Isomap 3. Lastly, classical MDS is used to extract the embedded lower dimensional space [18]. In the case of Fig 3, the embedded subspace is a two dimensional surface. As it turns out, while human motion is complex and multi dimensional, Isomap has been used successfully to simplify analysis and classification of human motion [5], [6]. See Fig 4, and 5 for examples of FSL signs and their corresponding Isomap manifold. We now have reduced complex, multidimensional signs into something easier to work with. 4. DYNAMIC TIME WARPING When FSL signs are performed there is considerable variation between samples, even when performed by the same person. Without affecting the meaning, signs can be performed quickly or slowly; parts of the sign may be performed at varying speeds. How can we compare data sets that vary in length and with portions that may be slightly faster or slower? Dynamic Time Warping (DTW) deals with exactly these issues. Figure 5: BRAVE sign and Isomap DTW is a non-linear mapping from one time series data to another; aligning two similar but locally out of phase

datasets. It has been successfully applied to various tasks such as classification and anomaly detection in time-series data, speech recognition and data mining [9],[10].

Starting from time t = 0, a contiguous path of elements in the distance matrix is calculated that minimizes the accumulated distance.

(see Fig 6) The warping path can be found using dynamic programming, evaluating the accumulated distance. w k 5. METHODOLOGY 5.

Each signer performed sixty 2-handed signs and fifty-seven 1-handed signs for a total of 117 unique FSL signs. Only traditional signs were used.

4 datasets. It has been successfully applied to various tasks such as classification and anomaly detection in time-series data, speech recognition and data mining [9],[10]. The DTW algorithm can be summarized as follows: 1. Given two time series data, Q and C, we construct a distance matirx. Euclidean distance between every other point is calculated and stored. 2. Starting from time t = 0, a contiguous path of elements in the distance matrix is calculated that minimizes the accumulated distance. Specifically, the warping cost is minimized: DT W (Q, C) = min K k=1 Where w k is the k th element of the warping path. (see Fig 6) The warping path can be found using dynamic programming, evaluating the accumulated distance. w k 5. METHODOLOGY 5.1 Data Collection Three native FSL signers were recorded individually while performing FSL signs. Each signer performed sixty 2-handed signs and fifty-seven 1-handed signs for a total of 117 unique FSL signs. Only traditional signs were used. Signers was seated in front of a plain, black background. The video camera was placed on a tripod approximately 160 cm away from the signer. Zoom was adjusted such that the signing space was captured. Two lights were placed approximately 160 cm away on either side to reduce shadows and uniformly illuminate the signers. All signers wore plain black, short-sleeved shirts. The video camera was set to record at full color, pixel count of 640x480 at 30 frames-per-second (fps). Each sign was performed in isolation, that is, with no context and not part of a sentence or discourse. The FSL sign was performed as close to the citation form as possible. We define the neutral position (Fig 8) to be arms on the side and hands on the lap with a blank facial expression and facing forward. Signers begin at the neutral position, perform the sign, and then return to the neutral position. Figure 6: (A) Q and C are similar but slighty out of phase (B) DTW matching the datasets (C) the warping path through the distance matrix To reduce computation effort, constraints are placed on the distance matrix; only elements of the distance matrix falling within the constraint are considered in the warping path. The two most common constraints used are the Sakoe-Chiba Band and the Itakura Parallelogram [9]. (a) Sakoe-Chiba Band Figure 7: DTW Constraints (b) Ikatura Parallelogram Figure 8: The neutral position The signs were recorded in groups of 10, with about 3 seconds of the neutral position in between signs. Thirteen groups of signs were recorded with some signs appearing in more than one group. 5.2 Pre-Processing and Editing Video was scaled to 160x120 pixels, and converted to grayscale. Converting the video to grayscale simplifies the representation. Each pixel carries only the intensity information. Simple background subtraction was done by setting any pixel below a threshold to be black. This removes most of the background and foreground (the shirt of signer) leaving only the head and arms. 5.3 Training and Testing The Isomap manifolds of the signs were generated and stored. The manifolds are zero-centered about the mean and normalized by the standard deviation to simpify comparisons. Input to Isomap were either the individual signs (3-5 second clip) or a group of signs (60-70 second clip). For individual signs, the original video was edited to contain only one sign.

(a) original (b) pre-processed Figure 9: Image Pre-processing For a group of signs, the original video was edited to contain 10 signs in sequence; each sign separated by approximately 2 seconds of

These manifolds are then used as input to DTW for matching. We used the Sakoe-Chiba Band, one of the most commonly used constraint in DTW with 10% constraint as suggested by literature. [9].

Fig 11 show the S values of the LOLO manifold compared to other manifolds. The labels MJ, RM or RW indicate which of the three native signers performed the sign.

Our result mirrors the problem discovered by other sign language recognition research: sign language recognition across different signers is error prone.

5 (a) original (b) pre-processed Figure 9: Image Pre-processing For a group of signs, the original video was edited to contain 10 signs in sequence; each sign separated by approximately 2 seconds of the signer in a neutral position. Isomap can use either K-nearest neighbor or epsilon neighborhood. We chose K-nearest neighbor. The value of k=10 used was obtained through experiments. These manifolds are then used as input to DTW for matching. We used the Sakoe-Chiba Band, one of the most commonly used constraint in DTW with 10% constraint as suggested by literature. [9]. Accumulated distance (S) is normalized over the length of the warping path; low values of S indicate a close match with a value of zero (0) indicating an exact match. Fig 11 show the S values of the LOLO manifold compared to other manifolds. The labels MJ, RM or RW indicate which of the three native signers performed the sign. Figure 11: Sample S values for LOLO maximum of 1.33 and a minimum of Our result mirrors the problem discovered by other sign language recognition research: sign language recognition across different signers is error prone. This is explained partially by the dataset used, a large portion of the dataset of which consists of minimal pairs. In sign language linguistics, a minimal pair is a pair of signs that differ in only one parameter (see 2.2). Minimal pairs, as defined, are already very similar, possibly to the point where it leads to false positives. 7. CONCLUSION In this paper, we described a recognition system for FSL based on Isomap manifolds. Our significant finding is that Isomap is good at discriminating large arm and body movements and weak at detecting hand shape against the large movement of the arms and body. This implies that Isomap manifold-based recognition requires additional processing for the analysis of hand shape and facial expression. Figure 10: Example warping path for BRAVE match 6. RESULTS The same FSL sign, performed by different signers, the average S value from DTW is 1.11 with σ = 0.67; with a maximum of 1.52 and a minimum of Different FSL signs, performed by different signers, the average S value from DTW is 1.28 with σ = 0.03; with a 8. REFERENCES [1] Rafaelito M. Abat and Liza B. Martinez, The History of Sign Language in the Philippines: Piecing Together the Puzzle, In 9th Philippine Linguistics Congress, 2006, Diliman, Quezon City [2] Yvette S. Apurado and Rommel L. Agravante The Phonology and Regional Variation of Filipino Sign Language: Considerations for Language Policy, In 9th Philippine Linguistics Congress, 2006, Diliman, Quezon City [3] Julius Andrada and Raphael Domingo, Key Findings For Language Planning From The National Sign Language Committee (Status Report On The Use Of Sign Language In The Philippines), In 9th Philippine Linguistics Congress, 2006, Diliman, Quezon City [4] Marie Therese A.P. Bustos and Rowella B. Tanjusay, Filipino Sign Language in Deaf Education: Deaf and

6 [12] Sylvie C.W. Ong and Surendra Ranganath, Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning, IEEE Trans. Pattern Analysis & Machine Intelligence, June 2005 no.6, vol.27, pp [13] Phil. National Statistics Office, Persons with Disability Comprised 1.23 Percent of the Total Population, Special Release No. 150, March [14] Phil. Deaf Resource Center and Phil. Federation of the Deaf, Part 1: Understanding Structure An Introduction to Filipino Sign Language, 2004, Phil. Deaf Resource Center [15] Phil. Deaf Resource Center and Phil. Federation of the Deaf, Part 2: Traditional and Emerging Signs, An Introduction to Filipino Sign Language, 2004, Phil. Deaf Resource Center [16] Christian Philipp Vogler, American Sign Language Recognition: Reducing the Complexity of the Task with Phoneme-Based Modeling and Parallel Hidden Markov Models, PhD thesis, University of Pennsylvania, 2003 Figure 12: FSL recognition flowchart Hearing Perspectives, In 9th Philippine Linguistics Congress, 2006, Diliman, Quezon City [5] Jaron Blackburn and Eraldo Ribeiro, Human Motion Recognition Using Isomap and Dynamic Time Warping, In Second Workshop, Human Motion, Oct 2007, Rio de Janeiro, Brazil Lecture Notes in Computer Science 4814, pp [6] Heeyoul Choi, Brandon Paulson, and Tracy Hammond, Gesture Recognition Based on Manifold Learning, Lecture Notes in Computer Science 5342, pp [17] World Health Organization, Deafness and Hearing Impairment, Fact Sheet N300, March 2006, World Health Organization [18] Joshua B. Tenenbaum and Vin de Silva and John C. Langford, A Global Geometric Framework for Nonlinear Dimensionality Reduction, Science, Dec 2000, no.5500 vol.290 pp , isomap [19] Sam T. Roweis and Lawrence K. Saul, Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds, Science, Dec 2000, no.5500 vol.290 pp , [7] Philippe Dreuw and Carol Neidle and Vassilis Athitsos and Stan Sclaroff and Hermann Ney, Benchmark Databases for Video-Based Automatic Sign Language Recognition, In International Conference on Language Resources and Evaluation, May 2008, Marrakech, Morocco, dreuw/database.php [8] Raymond G. Gordon Jr. (editor), Ethnologue: Languages of the World, 15th ed., SIL International, 2005, Dallas, Texas, [9] Chotirat Ann Ratanamahatana and Eamonn Keogh, Three Myths about Dynamic Time Warping, In SIAM International Conference on Data Mining, April 2005, Newport Beach, CA [10] Eamonn Keogh and Michael Pazzani Scaling up dynamic time warping to massive datasets, In 3rd European Conference on Principles and Practice of Knowledge Discovery in Databases, 1999, Prague, Czech Republic [11] Liza B. Martinez, Personal Communication, June 2008

Real Time Sign Language Processing System

Real Time Sign Language Processing System Dibyabiva Seth (&), Anindita Ghosh, Ariruna Dasgupta, and Asoke Nath Department of Computer Science, St. Xavier s College (Autonomous), Kolkata, India meetdseth@gmail.com,