Utilizing Bio-Mechanical Characteristics For User-Independent Gesture Recognition

Transcription

1 Utilizing Bio-Mehanial Charateristis For User-Independent Gesture Reognition Farid Parvini, Cyrus Shahabi Computer Siene Department University of Southern California Los Angeles, California Abstrat We propose a novel approah for reognizing hand gestures by analyzing the data streams generated by the sensors attahed to the human hands. We utilize the onept of range of motion in the movement and exploit this harateristi to analyze the aquired data. We show that sine the relative range of motion of eah setion of the hand involved in any gesture is a unique harateristi of that gesture, it provides a unique signature for that gesture aross different users. Based on this observation, we propose our approah for hand gesture reognition whih addresses two major hallenges: user-dependeny and deviedependeny. Furthermore, we show that our approah neither requires alibration nor involves training. We apply our approah for reognizing ASL signs and show that we an reognize stati ASL signs with no training. Our preliminary experiments demonstrate more than 75% auray in sign reognition for the ASL stati signs. 1. Introdution Human motion reognition has many important appliations suh as improving human-omputer interation in the virtual reality appliation domain. This researh area onerns the traking, detetion and reognition of the movement of people and more generally, understanding the human behavior. Due to the diversity of human sizes and ergonomi measures, one of the most hallenging issues in this ontext is reognizing the movement of different people robustly regardless of this diversity. A more speifi prob- This researh has been funded in part by NSF grants IIS (PECASE) and IIS , and unrestrited ash gifts from Mirosoft. Any opinions, findings, and onlusions or reommendations expressed in this material are those of the author(s) and do not neessarily reflet the views of the National Siene Foundation. lem arises during gesture reognition while various users making the same gesture. In order to reognize the movement or speifially a gesture, the user is traed and monitored through various sensory devies suh as traking devies on her hands or hapti devies. Data olleted from the sensors of these devies are onsidered as Multi-Stream Human Sensor Data, or MSHSD as we all, whih is the ontinuous immersive data stream whih is generated by the sensors attahed to human body. This data type has the following speial harateristis, it is: user-dependent devie-dependent noisy User-dependeny implies that the data generated by different users for the same experiment are not idential. Calibration is the proess that devie manufaturers suggest to make the generated data as idential as possible. It is the omparison of a measured value of unverified auray to a verified auray measure to detet any variation from the required performane speifiation. Mahine learning tehniques (e.g., neural networks, deision trees or ase-based reasoning) are also addressing user-dependeny. They are the algorithms that generate a model based on the previously seen data (i.e., training data) in order to lassify the new data. Another relevant hallenge in gesture reognition is devie-dependeny. That is the generated data by two different devies for the same experiment are ompletely different. In this paper, we propose an approah for reognizing gestures based on the bio-mehanial harateristis of the movement of the hand. In our approah, we abstrat out a unique signature for eah sign based on the range of motion of the sensors (joints) involved in that gesture. Our researh is distint and novel in the following three aspets. First, to the best of our knowledge,

2 Sensor values : s1,,sn T I M E ; ; ; ; ; ;251265; ; ; ;0.9114; ; ; ; ;0265; ; ; ; ; ; ; ; ;0.205; ; ; ; ; ; ; ; ;0.54; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ; ;0.5; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ; ;0.7; ; ; ;0.9114; ; ; ; ;0.0777; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ; ;0.777; ; ; ; ; ; ; ;0.1035;0.005; ; ; ; ; ; ; ;0.1035;04777; ; ; ; ; ; ; ;0.1035;04777; ; ; ; ; ; ; ;0.1035;04777; ; ; ; ; ; ; ;0.1035;077; ; ; ; ; ; ; ;0.1725;0777; ; ; ; ; ; ; ;0.1035;077; ; ; ; ; ; ; ;0.1035;04777; ; ; ;0.8463; ; ; ;0.1035;0.077; ; ; ; ; ; ; ;0.1035;077; ; ; ; ; ; ; ;0.0345; ; ; ; Figure 1. An illustration of a sample data set our approah in utilizing bio-mehanial harateristis is unique among all the studies who have intended to analyze the olleted raw data for gesture reognition. Seond, our approah addresses the major hallenges involved in analyzing MSHSD: user-dependeny, devie-dependeny and noisy data. Finally, our approah does not require any sort of training or alibration, a must in most mahine learning based approahes for gesture reognition. Finally, while the fous of this paper is on the problem of lassifiation, our approah has broader appliability. With lassifiation, eah unknown data sample is given the label of its best math among known samples in a database. Hene, if there is no label for a group of samples, the traditional lassifiation approahes fail to reognize these input samples. For example, in a virtual reality environment where human behavior is aptured by sensory devies, every behavior (e.g., frustration or sadness) may not be given a lass label, sine they have no lear definition. Our approah an address this issue by finding the similar behavior aross different users without requiring to have them labelled. The remainder of this paper is organized as follows. Setion 2 explains the motivating appliation. Setion 3 formalizes the problem of analyzing data for gesture reognition. Setion 4 disusses the related work. We present our approah in Setion 5. The results of our experiments are reported in Setion 6. Finally, Setion 7 onludes this paper and disusses our future researh plans. 2. Motivation To motivate our researh of utilizing bio-mehanial harateristis for gesture reognition, we fous on reognizing Amerian Sign Language (ASL) signs as an example of a well-defined set of hand motions. Unlike general gestures, sign languages are highly strutured, whih makes Figure 2. ASL alphabets M, N, E and O the reognition problem easier, sine their strutures an be used to form abstration and exploit ontext. Thus, sign language reognition provides a good starting point for studying a more general problem of gesture reognition. In addition, a funtional sign language reognition system ould failitate the tedious proess of transribing onversations for sign language researh tremendously, as well as failitating the interation between deaf and hearing people. Figure 3. The signing of the word time in ASL whih onsists of the individual alphabets T, I, M and E Amerian Sign Language (ASL) is a omplex visualspatial language that is used by the deaf ommunity in the United States and English-speaking parts of Canada. It is a linguistially omplete and natural language. Some people have desribed ASL and other sign languages as gestural languages. ASL also has the advantage of inluding two types of gestures, stati and dynami, hene it is a perfet appliation for investigating our approah addressing different hallenges involve in reognizing both of these types. The main hallenges of our approah are analyzing the data and reognizing the gesture or gestures a user makes in realtime. We show that while our approah is user and devie independent, it requires neither training nor alibration. This distinguishes our work from other studies and we address this in more details in Setion Formal Definition In this setion, we formally desribe the problem and define the notations we use through out the paper. As we mentioned, the proess of gesture reognition starts with olleting data from the sensors attahed to the

3 hand of a user. At eah sensor lok, the sensory devie driver aptures one sample by aquiring data from all of the n sensors of the devie. Eah sample is stored in a tuple with n fields where eah field is assoiated with one sensor. We represent a sample at time t by S t = (s 1, s 2,..., s n ), where eah s i is a real number indiating the value that is aquired from sensor i at time t. As time evolves, a data set of samples is aquired and generated. An example of suh a data set of samples is shown in Figure 1. T I M E Sensor values : s1,,sn ; ; ; ; ;056; ; ; ; ; ; ; ;-1.066; ;-0.;0.0753; ; ; ; ; ; ; ; ;044;0.0753; ; ; ; ; ; ; ; ;0.;0.0753; ; ; ; TRANSITION ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;0; ; ; ; ; ; ; ; ; ;0; ; ; ; ; TRANSITION ; ; ; ; ;0.631; ; ; ; ; ; ; ;01631; ; ; ; ; ; ; ;0.631; ; ; ; ; ; ; ;01631; ; ; TRANSITION ; ; ; ; ;056;0.0753; ; ; ; ; ; ; ; ;044;0.0753; ; ; ; ; ; ; ; ;068; ; ; ; ; ; ; ; ; ;0;0.0753; ; ; ; TRANSITION No meaning No meaning No meaning No meaning T I M E 1. Stati Signs: These are the signs that aording to ASL rules, no hand movement is required to generate them. All ASL alphabets exluding J and Z are stati signs. In spite of the fat that one sample should be enough to represent eah alphabet, due to the nature of data aquisition proess, we require to represent eah stati sign with several samples. Hene we represent eah sign with S [t0,t] = {S t0,..., S t } j [t 0, t], S j A i, where A i represents an ASL alphabet in whih i {A,..., Z} {J, Z}. Ideally, for a stati sign, all the samples in the time period of t = t t 0 should be idential. However, sine the data aquired from the sensors are usually noisy and some minor movements are inevitable, the data set has some deviation. In other words, ignoring some unlikely exeptions, the onsequent samples in t would not be idential, but mostly their distanes from eah other are bounded by a threshold and then an be onsidered equal. Hene, we represent a stati sign more aurately by S [t0,t] = {S t0,..., S t } j [t 0, t] (Distane(S j, A i ) ɛ S j A i ). Figure 4. Data samples representing the finger-spelling of the word TIME in ASL In general, we represent the data set, whih is a olletion of samples, with C = {S t0,..., S t }, where t 0 denotes the starting time, t the ending time and t = t t 0 the period of sampling, respetively. Eah S i is the sample we olleted at time i. We also show the olletion of our data with an interhangeable notation as C [t0,t]. Sine eah sample an be onsidered as a point in an n-dimensional spae, we define two samples equal if their distane aording to a metri riteria is less than a threshold. In other words: S i S j Distane(S i, S j ) ε. The most straightforward approah for measuring the similarity between two samples is using a Minkowski measure suh as the Eulidean distane. Given two samples S i = (s 1,..., s n ) and S j = (s 1,..., s n), the Eulidean distane between S i and S j is defined as: Distane(S i, S j ) = ( i=1n si s i 2 ) 1/2 (1) We ompare the result of applying different distane metris in Setion 6. Considering this notation for eah sample, we aordingly lassify the ASL signs into the following two ategories: Figure 5. The sign for yellow is made by forming the letter y with the right hand in a irular movement 2. Dynami Signs: In ontrary to stati signs, the generation of these signs require movement of fingers, hand or both. We divide this group of signs into two subategories: Type I: These signs onsist of a series of different stati signs with some transitional movements between eah pair. The movements involve in these signs do not onvey any meaning by themselves and are just the transitions between different stati signs. The ASL finger-spelling words fall in this ategory. We represent a dynami sign type I with D [t0,t], where the time duration of t = t t 0 should be onsidered as a superset of some smaller durations eah of whih represents a stati sign. That is : [t 0, t] = [t 0, t 1 ] [t 1, t 1 + t 1 ] [t 1 + t 1, t 2 ] [t 2, t 2 + t 2 ]... + [t n, t] where C [t1,t 1 + t 1 ] represents the first stati sign and C [tn,t n + t n ] represents the nth stati

4 sign. With this notation, [t 1 + t 1, t 2 ] denotes the time period for transitional samples whih represents transitional movement between the first and seond stati sign and does not onvey any meaning by itself. An example of this type of dynami signs is shown in Figure 3 and its olleted data is shown in Figure 4. Type II: The rest of ASL dynami signs fall in this ategory. They an either represent an alphabet, i.e., J or Z or onvey a word, e.g., yellow. An example of this type of signs is shown in Figure 5. We represent this type of signs with (D [t0,t], M), where M represents the movement information in the sign. This movement is either the result of the movement of the hand, e.g., a stati sign in movement or is the result of both fingers and hand moving simultaneously, e.g., another dynami sign in movement. In this paper, we address the problem of reognizing the stati signs as well as the dynami signs of type I. The reognition of the dynami signs of type II involves two steps: 1. Reognizing the stati part (if any), and 2. Reognizing the movement of the fingers and/or the hand While we address the former in this paper, the latter is the fous of our future researh. Consequently, the elaboration of M is beyond the sope of this paper. The reognition of a stati sign requires finding the best math (i.e., the minimum distane) between a sample in C[t 0, t] (or D[t 0, t] for a dynami sign type I) and a known sample A i. In Setion 5, we will show that this problem annot be addressed by a simple searh and omparison. Before proeeding to desribe our approah for sign reognition, we survey the studies related to our work in the following setion. 4. Related Work Sign reognition has been studied extensively by different ommunities. We are aware of two major approahes: Mahine-Vision based approahes whih analyze the video and image data of a hand in motion and Hapti based approahes whih analyze the hapti data reeived from a sensory devie (e.g., a sensory glove). Due to lak of spae, we refer the interested readers to [12] for a good survey on vision based sign reognition methods. Within the hapti approahes, the movement of the hand is aptured by a hapti devie and the reeived raw data is analyzed. In some studies, a harateristi desriptor of the shape of the hand or motion whih represents the hanges of the shape is extrated and analyzed. Holden and Owens [5] proposed a new hand shape representation tehnique that haraterizes the finger-only topology of the hand by adapting an existing tehnique from speeh signal proessing. Takashi and Kishino [11] ould reognize 34 out of 46 Japaneese kana alphabet gestures with a data-glove based system using joint angle and hand orientations oding tehniques. Newby [8] used a sum of squares template mathing approah to reognize ASL signs. Hong et. al [6] proposed an approah for 2D gesture reognition that models eah gesture as a Finite State Mahine (FSM) in spatial-temporal spae. More reent studies in gesture reognition have foused on hidden Hidden Markov Model (HMM) or Support Vetor Mahines (SVM), whih have produed highly aurate systems apable of handling dynami gestures [13]. While lassifiation an be aomplished by these model-based learning approahes, they fail to math the signs whih have no labels or detet the similar unknown signs. Many researhers proposed using neural networks to address the sign reognition problem. Murakami and Taguhi [7] use reurrent neural networks to detet signs from Japanese Sign Language. Boehm et. al [2] proposed using neural network for dynami gesture reognition. The dominant distintion between all these approahes and ours is that they rely on a training phase and that requires a lot of training data. In addition, the results of these tehniques are diretly affeted by the data set hosen for the training phase and onsequently they are userdependent. The idea of devie independeny has been addressed in a number of studies. Su and Furuta [10] propose a logial hand devie that is in fat a semanti representation of hand posture. This was done with the express purpose of ahieving devie independeny, but to our knowledge it was never implemented. 5. ROMAS: Range Of Motion Abstration for Sign reognition Our solution to reognize a sign is based on the observation that all forms of hand signs inlude finger-joint movements from a starting posture to a final posture. To abstrat this movement, we utilize the onept of range of motion from the Bio-Mehanial literature [9] at eah joint. Range of Motion (ROM) is a quantity whih defines the joint movement by measuring the angle from the starting position of an axis to its position at the end of its full range of the movement. For example, if the position of a joint axis hanges from 20 to 50 with respet to a fixed axis, the range of motion for this joint is 30. We ompute the range of motion per joint by using the sensor values aquired by the sensory devie. The main intuition behind our approah is that the range of motion of eah setion of the hand partiipating in a sign, relative to the non-partiipating setions,

5 is a user-independent harateristi of that sign. This harateristi provides a unique signature for eah sign aross different users. We now disuss our approah in more details. a sign, respetively. We alulate the range of motion tuple R t as follows : R t = S t S t0 = (r 1,..., r n ) i (1,..., n), r i = s i s i and t = t t 0 Figure 6. Starting Posture and final Posture Suppose that a user U is making the stati sign L by wearing a sensory devie. The user is required to start making the sign from a starting posture toward a final posture. An example of one possible starting posture and the posture representing sign L are shown in Figure 6. Note that our tehnique does not rely on a speifi starting posture, in fat, any starting posture will work for us, as long as it is onsistent through all experiments. The samples assoiated with these two postures an be represented as S t0 and S t, respetively. For the user U and sign L, the values of the samples may be similar to S t0 = ( ,..., ) and S t = ( ,..., ). The sensory devie and what eah sensor value represents are explained in more details in Setion 6. If the user repeats making the same sign, due to noisiness of data and some inevitable movements, the raw data would be different as ompared to the first experiment. In addition, this raw data is ompletely user dependent, i.e., the tuples generated by users U and U making the same sign are ompletely different. Due to these irumstanes, having an exat math (i.e., an idential sample) for an unknown sample among existing samples is almost impossible. Consequently, reognizing a sign by attempting to searh through existing samples in order to find an exat math is not possible either. The best that an be ahieved by searhing is finding a sign with distane less than a threshold. As it was mentioned before, this approah also fails due to the diversity of raw data. Hene, our objetive is to transform the diverse raw data sample to an abstrated unique sample. This transformation ompensates the dissimilarity of the olleted data and provides a unique signature aross different experiments for the same sign (sign L in this ase). Towards this end, we introdue ROMAS, the algorithm that provides this transformation by utilizing the onept of range of motion. The pseudo-ode for this algorithm is shown in Figure 9. Suppose S t0 = (s 1,..., s n ) and S t = (s 1,..., s n) represent the samples of the start and final postures for making Figure 7. ASL sign L and its representation with k=4 For example, for sign L, R t is alulated as ( ( ),..., ( )) or ( ,..., ) R t is a tuple onsisting of n positive or negative real numbers depending on the diretion of the movement. In order to normalize the values of the movement, we first onstrut R = ( r 1,..., r n ). The rationale behind using absolute values is that smaller values (i.e., larger negative numbers) in R do not neessarily mean less movement. To apture the diretion of the movement and differentiate between movements in opposite diretions, for eah R, we alulate D whih holds diretional information as follows: D t = (d 1,..., d n ) where { 1 if ri 0 d i = 1 otherwise Figure 8. ASL sign F and its representation with k=4 Subsequently, we find the maximum and minimum values within R and represent them with M(R) and m(r),

6 respetively. We then normalize eah value in R by subtrating m(r) and dividing the result by (M(R) m(r)). We represent the result of this normalized R whih onsists of values between 0 and 1 with NR. Finally, we disretize the values of NR with a given disretization parameter k(> 1). For example, if k = 2, we replae eah value of NR with 0 if its value is less than 0.5, and 1 otherwise. The resulting NR for sign L with k = 4 is : NR = ( 0.25; 0.25; 0.25; 0.25; 0.25; 0.25; 0.25; 0.25; 0.75; 1.0; 0.5; 0.25; 0.75; 1.0; 0.5; 0.25; 0.75; 1.0; 0.5; 0.25; 0.5; 0.25). While this tuple ontains valuable information about relative movement of eah sensor with respet to others for sign L, it does not provide any information regarding the diretion of the movements. Multiplying eah item of NR by its orrespondent item in D results a new tuple NRD = ( n 1,..., n n ) i {1,..., n}, n i = n i d i whih provides both the diretion and the relative value of the movements. The alulated NRD for sign L is shown as follows: NRD =( -0.25; -0.25; 0.25; 0.25; -0.25; 0.25; 0.25; -0.25; -0.75; -1.0; -0.5; 0.25; -0.75; -1.0; -0.5; 0.25; -0.75; -1.0; - 0.5; 0.25; 0.5; 0.25). Sine N RD represents the harateristi of the movement of the sensors making a partiular sign, it provides an abstration for that sign. We all this abstration the signature of the sign and observe that while the signature is unique for eah sign, it is idential among different users making the same sign. That is, if different users wear the sensory devie and make a speifi ASL sign, while the raw data generated by the sensors are ompletely different, the alulated N RDs are almost idential aross all of them. This also implies that by abstrating the sign with its signature, we eliminate the effet of inevitable noise produed by the sensors during the data olletion proess. The uniqueness of this signature provides us with the very important property of user independeny. For k = 4, we visualize the resulting NRD with the following oding for eah sensor: White olor (no mark) if n i [0, 0.25] Yellow olor (+ mark) if n i [0.25, 0.5] Orange olor (- mark) if n i [0.5, 0.75] Red olor (X mark) if n i [0.75, 1.0] Figures 7 and 8 show the ASL representation of sign L and F, respetively, with our sensor value oding for k = 4. In the following setion, we disuss our method for reognizing ASL signs based on ROMAS Reognizing ASL signs by ROMAS In this setion, we first present our approah for reognizing the ASL stati signs and then extend our approah 1 Reognize Stati Sign( S i, k, (A 1,..., A n) { 2 S t0 := Sample of Starting Time 3 while( S i S t ) { 4 R i = S i S t0 ; 5 R i = R i ; 6 CalulateD i; 7 M(R) = Maximum(R i ); 8 m(r) = Minimum(R i ); 9 NR i = R i m(r) (M(R) m(r)) 10 NR i = Disretisize(NR i, k); 11 NRD i = NR i D i for (j = 1 to n) { 14 if Distane(NRD, NRD(A j)) ɛ 15 return A j 16 } 17 } 18 Return null Figure 9. ROMAS Algorithm for reognizing a stati sign to show how to reognize a dynami signs type I. Finally we address the problem of finding similar movements without labelling them. In order to reognize an unknown stati sign made by a user, we require to ompare its signature with the signatures of some known samples. Consequently, the first step is olleting the data for eah stati sign one and alulating its orresponding N RD. We all this proess registration and save all the registered signs in our registration database. The registration is ompletely different from training in the following aspets: In ontrast to training that requires several sets of data, we require one set of samples (one sample for eah sign) to ompletely register all signs. Registration is user-independent, i.e., any user an register any sign while ombinations of users are also aeptable. Registration is devie-independent, i.e., a sign an be registered with another devie or even without any sensory devie. The reason is that the information we require to register a sign is the relative movements of the sensors, if this information an be provided by another devie or even without using sensory devie, we still an register the sign. Registration is extensible, that is if we want to reognize some new signs, we an register them without having any effet on previously registered signs. While with some training based approahes, adding new signs requires training with all signs again, in our approah, we just register new signs and there is no requirement to register previous signs.

7 A u r a y Manhatan Eulidian Fuzzy 1 Canberra Distane Measure Figure 10. Comparison of different distane measures To reognize an unknown sign, we identify the best math for that sign in our database. We first ollet its sample data and alulate its orresponding N RD. The only requirement is that the starting posture for making this unknown sign should be idential with the starting postures of previously registered signs. We then ompute the distane between the alulated NRD of the unknown sign and NRDs of all registered signs in the database and find the one whih has the least distane. If the distane is ɛ, the unknown sign is onsidered idential with the one with the minimum distane. A u r a y k=2 k=4 k=6 k=8 K = 10 Disretization Parameters Figure 11. The impat of the disretization value, k It is neessary to mention that sine we are utilizing a high-level semanti data rather than low-level sensor values, it is possible to employ simpler and more extensible approahes for finding the best math (i.e., minimum distane). It is also possible to take advantage of multidimensional indexing strutures(e.g., R-Tree) to expedite the searh proess. We have experiened with several distane measures metris and found that the Eulidean distane measure provides the fastest and most aurate result (refer to Setion 6). We now move forward and apply our method to address a more hallenging problem, reognizing dynami signs of type I. In order to do so, we require to identify a series of stati signs in the data set. Towards this end, we follow the same proedure as we did in the proess of stati sign reognition, that is for eah oming sample, we alulate its orresponding NRD and find the best math ompared to the registered signs. If there exists any NRD with distane less than ε from the unknown NRD, we onsider it a math and save it in an output buffer. Sine all samples in C [ti,t i + t i ] [t i, t i + t i ] [t 0, t 0 + t] are representing one stati sign, we utilize this buffer to disard repeating signs and onsider only one sign for eah time period of [t i, t i + t i ]. Hene for eah set of samples within [t i, t i + t i ], we generate one sign in the output buffer. We observed that it is not required to alulated NRD for eah oming sample. Our experiments demonstrate that as long as one sample within [t i, t i + t i ] is reognized, we ahieve the same auray as alulating NRD for all samples. In order to find similar gestures without having them labelled, we onsider the ase that a large amount of signers make some unknown stati signs whih do not belong to ASL. Sine these signs have no label, we are not able to reognize them by traditional sign reognition tehniques. However, in our approah, we an generate a database whih keeps all the signs along with their orresponding signatures. To find the similar mathes for eah sign of interest, we searh through the database and find the samples with the the minimum distanes. We an find the best math (nearest neighbor) among signatures or find several mathes (k-nearest neighbors). We also an utilize various indexing tehniques to perform this task faster. In the next setion, we present the result of our experiments. 6. Performane Results In this setion, we present the results of our experiments that we onduted to evaluate our approah in reognizing ASL stati signs. The objetives of our experiments were: 1. Speifying the best values for our parameters (e.g., k) to ahieve the highest possible auray and determining this auray (i.e., the number of orretly reognized signs out of 24). 2. Comparing our approah with a onventional approah utilizing neural network. We will show that our approah ahieves at least a omparable performane to the onventional approah, while providing a level of user-independeny and devie-independeny that is beyond the apabilities of any onventional approah. We first explain the setup that we used for our experiments and then present the results.

8 Signs registered by signer 3, K = 6 A B C D E F G H I K L M N O P Q R S T U V W X Y Reognized Auray Signer 1 A B E D E O L H I K L N N O P Q K S Q U U W X Y % Signer 2 A B E D E F G R I K L M M A P T R S T U U W X Y % Signer 3 A B C D E F G H I K L M N O P Q R S T U V W X Y % Signer 4 A B C D E F X K I K L M A E P Q R S T R V W X Y % Signer 5 A C R E F G P I H L M R O G T R S T U V W X Y % Signer 6 A B C D E F L P I K L M M O P G K S T U P W X Y % Signer 7 M B E D M F G K I K L M U O P L R S T U R W X Y % Signer 8 A B C X E F D H Y K L M N D P T R M T U V W S Y % Signer 9 A B C D E F X H I K L U L C P T R E Q U H W X Y % Signer 10 A B C D E F G H I H G M N O T V R S X V R W X Y % 9-A 9-B 7-C 8-D 9-E 9-F 5-G 5-H 9-I 8-K 9-L 8-M 4-N 6-O 8-P 3-Q 8-R 8-S 7-T 8-U 4-V 10-W 9X 10Y Total 75.00% 1-M 1-3-E 1-X 1-M 1-O 2-L 2-K 1-Y 2-H 1-G 1-N 2-M 1-D 1-T 4-T 2-K 1-M 2-Q 1-R 2-U 1-S 1-R 2-X 2-P 1-U 1-A 1-A 1-G 1-G 1-E 1-X 1-U 2-R 1-D 1-R 1-R 1-E 1-L 1-P 1-U 1-C 1-V 1-H 1-L Figure 12. Comprehensive result for the first set of experiments with k=4 and Eulidean distane measure 6.1. Experimental Setup For our experiments, we used CyberGlove [1] as a virtual reality user interfae for aquiring data. CyberGlove is a glove that provides up to 22 joint-angle measurements. It uses proprietary resistive bend-sensing tehnology to transform hand and finger motions into real-time digital jointangle data. This glove model has three flexion sensors per finger, four abdution sensors, a palm-arh sensor, and sensors to measure flexion and abdution. A piture of this glove and the loation of eah sensor is shown in Figure 8. We initiated our experiments by olleting data from one signer wearing the glove and making all stati signs ( A to Y, exluding J ) from the starting posture as shown in Figure 6. For eah sign, we olleted 140 tuples from the starting posture to the final posture, eah inluding 22 sensor values. We registered all these signs for this signer and stored the data to be used in the next phase. In the next step, we olleted the same amount of samples (140 tuples, 22 sensors) from ten different signers (inluding the original signer) wearing the glove and making all 24 stati signs. The olleted data were saved in 240 files, eah onsist of 140 samples aquired while making one stati sign. Sine the appliation that we implemented for our experiment is apable of replaying these stored data as on-line data streams, the olleted data would be suffiient for running multiple experiments with various parameters. During our experiments, we varied the disretization parameter k from 2 to 10. We also tested with 4 different distane metris. Finally, we ompared the auray of our approah with the result of a neural network based system. We now present the results of our experiments Experimental Results Clearly the performane of a similarity query (as a omponent of gesture reognition) is determined largely by the hosen distane metri. So in the first set of experiments, we tested four different distane measures: Eulidean, Manhattan, Canberra [4] and fuzzy [3]. We read all 240 files one by one to reognize the sign. For this experiment, we hose (k = 6) as the disretization parameter and alulated the distanes between samples based on eah of these distane metris. The result of this set of experiments is shown in Figure 10. The result reveals that the Eulidean distane measure provides the best result with the highest auray of 75.5%.

9 In the seond set of experiments, we repeated the same experiments as the first set while varying the disretization parameters from 2 to 10 inrementing by 2. The result is shown in Figure 11. The figure illustrates that the auray improves by inreasing the disretization parameter from 2 to 4 and then to 6, but does not hange signifiantly over 6. The observation is that our normalization and disretization methods an improve the auray only to some extends. Investigating other normalization and disretization methods are the fous of our future studies. A u r a y ROMAS Neural Net 20 Considering these results, we repeated the same experiments with k = 6 and the Eulidean distane measure to find the overall auray of our approah. The result of this set of experiments is shown in Figure 12. In this figure, the result of the experiment for eah user is shown in one row. Two olumns in the right represent the number of orretly reognized signs and the orresponding auray, respetively. The table at the bottom of the figure shows what and how many were the signs reognized in 10 experiments for eah sign. The result of this experiment shows that our proposed approah is apable of reognizing most of the stati signs orretly and the signs whih were not reognized orretly are the very similar ones, e.g., M and N or E and O, as it shown in Figure 2. In the last set of experiments, we used a feed-forward bak propagation neural network from the Matlab neural network pakage. This network has one hidden layer with 40 neurons. The input layer has 22 neurons (eah for one sensor) and 24 neurons (one for eah sign) in the output layer. We performed the standard leave-one-out experiment with the subjets, that is, the network was trained with the data sets belonging to nine different users. The training data inluded the tuples onsisting of 22 values reeived from 22 sensors plus the minimum, maximum, average and median of the items in the tuple. We then tested for the 10th user who was exluded from trainees. We repeated the test 10 times, eah time left a different signer out of training and tested with her data set. After averaging, the neural net had the overall auray of 67%. We performed a different version of experiments, alled keep-one-in with ROMAS. That is we registered all signs with one user and then tested with the data sets belonging to 9 other signers. We repeated this test 10 times, eah time registered the signs with a different signer. The overall auray of ROMAS for this experiments was 75%. Though keep-one-in is onsidered a tougher test over leave-on-out (registration by one versus training with nine), the auray of ROMAS was higher than neural network approah. In Figure 13, the result of this omparison for eah sign is shown A B C D E F G H I K L M N O P Q R S T U V W X Y ASL Stati Alphabets Figure 13. Comparing ROMAS with Neural Network 7. Conlusion and Future work We proposed an approah for reognizing hand gestures based on the bio-mehanial harateristis of the movement of the hand during the formation of the gesture. Our approah has the following advantages over previous studies: It is user-independent It is devie-independent It does not require alibration There is no training involved Our approah is also apable of deteting the similar hand gestures without having them labelled, a problem that most traditional lassifiation methods fail to address. Our experimental results onfirm the effetiveness of our approah for reognizing both stati ASL signs and dynami signs type I. In our future researh, we plan to fous on the following issues: Investigating different methods of normalization and disretization to improve the auray. Approahing the reognition proess in two steps. In the first step, we will reognize the similar signs (e.g. A, C and S ) and then fine tune within the reognized group. Utilizing the onepts of Context, N-gram letter model and Ditionary to improve the auray of fingerspelling. Converting our numerial signature of the signs to some form of strings and then investigating the string mathing approahes for finding the similar math between signatures.

10 Reognizing the dynami sign of type II. Referenes [1] Immersion orporation, [2] K. Boehm, W. Broll, and M. Sokolewiz. Dynami gesture reognition using neural networks; a fundament for advaned interation onstrution. SPIE Conferene Eletroni Imaging Siene and Tehnology, San Jose, CA. [3] D. V. der Weken, M. Nahtegael, and E. Kerre. Some new similarity measures for histograms. Proeedings of ICVGIP 2004 (4th Indian Conferene on Computer Vision, Graphis and Image Proessing, Kolkata, India, Deember [4] S. M. Emran and N. Ye. Robustness of anberra metri in omputer intrusion detetion. [5] E. J. Holden and R. Owens. Representing the finger-only topology for hand shape reognition. Mahine Graphis and Vision International Journal, 12(2), [6] P. Hong, T. S. Huang, and M. Turk. Construting finite state mahines for fast gesture reognition. International Conferene on Pattern Reognition (ICPR 00), 3, [7] K. Murakami and H. Taguhi. Reognition using reurrent neural networks. Human Interfae Laboratory, Fujitsu Laboratories LTD. [8] G. B. Newby. Gesture reognition using statistial similarity. [9] N. B. Reese. Joint Range of Motion. ISBN , [10] S. A. Su and R. K. Furuta. Vrml-based representations of asl fingerspelling. Proeedings of the third international ACM onferene on Assistive tehnologies. [11] T. Takahashi and F. Kishino. Hand gesture oding based on experiments using a hand gesture interfae devie. pages 67 73, [12] Y. Wu and T. S. Huang. Vision based gesture reognition a review. International Gesture Workshop, GW 99, Frane, Marh [13] J. Yang and Y. Xu. Hidden markov model for gesture reognition. Tehnial report, Robotis Institute, Carnegie Mellon University, Pittsburgh, PA.