SPEECH TO FACIAL ANIMATION CONVERSION FOR DEAF CUSTOMERS

Transcription

1 SPEECH TO FACIAL ANIMATION CONVERSION FOR DEAF CUSTOMERS György Takács, Attla Thany, Tamás Bárd, Gergely Feldhoffer, Bálnt Srancsk Faculty of Informaton Technology, Péter Pázmány Catholc Unversty H 1083 Práter u. 50/a., Budapest, Hungary, phone: + (36) , fax: + (36) , emal: {takacs.gyorgy, thanya, bard, flug, sraba}@tk.ppke.hu. ABSTRACT A speech to facal anmaton drect converson system was developed as a communcaton ad for deaf people. Utlzng the prelmnary test results a specfc database was constructed from audo and vsual records of professonal lpspeakers. The standardzed MPEG-4 system was used to anmate the speakng face model. The traned neural net s able to calculate the prncpal component weghts of feature ponts from the speech frames. The control coordnates have been calculated from PC weghts. The whole system can be mplemented n standard moble phones. Deaf persons were able correctly recognze about 50% of words from lmted sets n the fnal test based on our facal anmaton model. 1. INTRODUCTION The prmary am of our project was to develop a communcaton ad for deaf persons whch can be mplemented n a moble telephone. In our system a partally anmated face s dsplayed n nteracton wth deaf users. The control parameters of the anmaton are calculated drectly from the nput speech sgnal. It s known that showng the face tself s a lmted representaton of the human speech process and contans nherent errors, though deaf people have fantastc abltes n understandng speech based on lp readng only. In spte of the lmtatons deaf persons aded wth specal software on the platform of hgh-end class second or thrd generaton moble phones could naturally communcate wth hearng people. KTH s Synface system [5] can successfully ad the communcaton of hard of hearng people by speech to anmaton converson. Synface shows the lp movements of the speaker at the other telephone synchronzed wth the speech sound. Our ntenton s to ad completely deaf people whch can be based on vsual modalty only. Speech to anmaton converson n Synface s dvded to a phoneme recognton module [7, 11] and a vsual speech syntheser [2], whch s drven by the phoneme strng. In our system only contnuous types of transformatons are used n the complete audo to vsual converson, no dscrete classfcaton method s appled. One of the benefts of our drect soluton s that the orgnal temporal and energy structure of the speech are retaned, so the naturalness of rhythm s guaranted. Further beneft s the relatvely easy mplementaton n moble phone envronments wth lmted memory and computaton power. A rather promsng feature of our system s the potentally language ndependent operaton. A very mportant element of our concept s to tran the system on a unque audo-vsual database collected from professonal nterpreters/lp-speakers. Ther artculaton style and level are adapted to deaf communcaton partners. One of the known soluton groups use anmated talkng faces to ncrease speech ntellgblty ganng addtonal nformaton related to the only audtory stuaton n cases of nosy envronment or hearng-mpared lsteners. Ths socalled superaddtve combnaton of audtory and vsual speech can produce a bmodal accuracy, whch s greater than the smple sum of ther separate unmodal performances [8, 9, 10]. In our applcaton only a unmodal communcaton s supposed owng to the deaf users. We focus on the vsual nput only and calculate wth the enhanced abltes of deaf persons. The dynamcs of mouth movements and the naturalness of face anmaton models seem to be crtcal parts of audoto-vsual converson. Usually researchers elaborate very sophstcated procedures to produce dynamc and natural talkng heads [3, 6]. We have selected the speakers for the data base recordng wth specal attenton to the hgh dynamc requrements. 2. DATABASE DESIGN AND COLLECTION 2.1 Prelmnary lp-readng tests Ths research study was started wth several lp-readng experments to measure the communcaton sklls of deaf people, and to understand ther everyday problems better. In ths paper only a bref summary of the results and conclusons s presented. One of our mportant conclusons was that vsual lpreadablty of the speech greatly depends on the qualty of artculaton. Lp-readng needs hgher level attenton to understand speech and msunderstandngs are more frequent. Therefore clear artculaton whch emphaszes dstnctve features and a slower speech rate can help a lot. The most lpreadable speakers wthn the hearng socety are nterpreters/lp-speakers. They have every day contacts wth deaf persons so they are able to adapt ther artculaton for lpreadng. Therefore we have decded to employ nterpreters to record our audovsual database. We also found that hearng mpared persons usually have dffcultes wth grammatcal rules. The context of tradtonal speech reference databases s too complex for them.

2 They often do not employ suffxes properly and are unable to follow sudden changes n the topc. They are usually unable to repeat complete sentences word by word. They keep only the essental parts of messages n ther mnd nstead. We have planned our audovsual speech database for both algorthm tranng and ntellgblty tests. The text materal of our database contans only two-dgt numbers, names of months, and days of the week. In the prelmnary tests the mportance of the thrd (z) dmenson (depth) of speakng faces were also tested. Two types of dstorted vdeos were presented to deaf subjects. In frst the blue component was ganed whle red and green were attenuated, and n the other type the pcture was bnarzed to black and whte on the brghtness of the pxels compared wth a threshold. In bnarzed vdeos the depth nformaton s almost completely hdden compared to the orgnal ones, and blue vdeos represent an ntermedate level n that sense. Surprsngly there was no sgnfcant dfference n recognton rates. Further experments were organsed wth small dsplays usng hand held moble phones and deaf customers recognzed well lp represented speech vdeo records. Ths led us to the concluson: only the area around the mouth s really mportant and enough to recognze speech. 2.2 Recordng Our database contans synchronsed audo and vdeo records from speakng persons. Audo and vdeo data are processed n synchronzed way; each vdeo frame has one audo frame. (see Fg. 1.) The head of speakers have been softly fxed to elmnate the moton of the head. Our system n the actual status s a speaker dependent soluton but we plan a lmted speaker ndependent verson. The MPEG-4 standard descrbes the face wth 86 feature ponts (FP). We selected 15 FP around the mouth accordng to our prelmnary results. Durng recordng the feature ponts were marked by yellow dots on the face of speakers. The advantages of MPEG-4 FP-s are the compressed descrpton of face parameters and the compatblty to standard facal anmaton models. Our speakers were professonal lp speakers Audo preprocessng Vdeo preprocessng Synchronsed audovsual Database Fgure 1 - Database collecton We used commercally avalable vdeo cameras wth 720x576 resoluton, 25 fps PAL format vdeo whch means 40ms for audo and vdeo frames. The vdeo recordngs have concentrated only on the area of the mouth and vcnty to let maxmum resoluton for FP extracton. The vdeo records were then processed automatcally. After de-nterlacng, we balanced the contrast, brghtness and saturaton of the vdeo records to enhance the marker ponts. The selecton of the marker ponts was based on RGB components. On the bnarzed, dlated and eroded pcture every spot covers only one pxel n the centre of the marker. Ths method has 1-2 pxel maxmum error Snce the average horzontal lattude of a FP-s are s 40-60, vertcally pxel, ths error s acceptable. The orgo has been chosen near to the nose, because ths FP-s (Fg. 2.) moves the least. The nput speech sound s sampled at 16 bt/48 khz. Fgure 2 - Selected subset of MPEG-4 feature ponts 3. CONVERTING SPEECH SIGNAL TO FACIAL ANIMATION Acoustc feature extracton NN PCA -1 MPEG4 based facal anmaton model MFCC w 1 w 6 weghts of the prncpal components FP Feature ponts coordnates Fgure 3 - Structure of the mplemented speech to facal anmaton system Our mplemented converson system s PC-based software. Here we survey the complete system at a glance, as t s shown n Fgure 3, and the detals of the buldng blocks are detaled n ponts

3 The nput speech sound s sampled at 16 bt/48 khz and then acoustc feature vectors based on Mel-Frequency Cepstrum Coeffcents (MFCC) are extracted from the sgnal. The feature vectors are sent to the neural network (NN), whch computes a specal weghtng vector [ w 1,...w 6 ] that s a compressed representaton of the target frame of the anmaton. The coordnates of our selected feature pont set - used to drve the anmaton - are obtaned by lnear combnaton of our component vectors wth the weghts comng from the neural net. Ths coordnate-recovery operaton s denoted by the term PCA -1 n the block dagram, because the predefned component vectors come from Prncpal Component Analyss (PCA). The FP postons are computed n ths way for 25 frames per second. (Fg. 5) The fnal component n our system s a modfed LUCIA talkng head model [4]. We control t wth the computed FP coordnates and then the facal anmaton model appears on the screen. See chapter Acoustc feature extracton The nput speech s pre-emphass fltered wth H(z)= z -1. Then 21.33ms long Hammng wndows are appled to the sgnal, and 16 Mel-Frequency Cepstrum Coeffcents (MFCC) are extracted from each analyss wndow. 5 analyss wndows are processed ths way for every frame of the anmaton, the mddle one s centred to the tme poston of the actual frame. The wndows are placed wth 40 ms dstance between them (Fg. 4). Fgure 4 - Acoustc nformaton from 5 wndows for each frame of anmaton Co-artculaton phenomenon has great mportance n both vsual speech and acoustcs, but n dfferent ways. Phonemes can be vsually domnant or flexble. The domnant ones have typcal vsual shape and hghly affect the fgure of the neghbourng flexble phonemes. The flexble ones tend to suffer the effects of the domnant neghbours. Analysng our database the closest relaton between acoustc and vsual parameters was found at the steady state part of the vsually domnant phonemes, these parts behave as anchors between the two modaltes. Durng flexble phonemes the relaton s much less determnate. In our experence t s advantageous f the converson algorthm accesses some acoustc nformaton from the steady state part of at least one of the neghbourng phonemes. When professonal lp-speakers talk to deaf people the speech rate falls down to 5-10 phoneme/sec. The steady state phases of domnant phonemes are emphaszed. Our 5 wndows partally cover about 180 ms, and lkely at least one of them reaches the quas-statonary phase of a domnant phoneme, whch provdes relable nformaton about the vsual shape. MFCCs from the 5 consecutve wndows are sent to the nput layer of the artfcal neural network (ANN). 3.2 Neural Network Matrx Back-propagaton algorthm (elaborated by Davd Anguta) was used to tran the network [1]. The NN has 3 layers: 80 elements n the nput layer to receve 16 MFCC from 5 frames. The hdden layer conssts of 40 nodes. The output layer has 6 nodes, provdng 6 centered PCA weghts to reproduce the FPs. The tranng fle contaned 5450 frames. The network was traned by epochs. Ths NN model uses nterval [-1,1] both for nputs and outputs. MFCC and PCA vectors are scaled lnearly by the same value to ft to the approprate range except the energy coeffcent of MFCC. The program of the traned neural network runs very fast n the workng system, snce all the database s represented n the weghts of the network, consstng from 3440 coeffcents only. So the calculaton has constant and tolerable tme consumpton and realsable by moble phones. Fgure 5 - The x-y components of FP-s as a functon of tme pronouncng word September. The upper sold lne shows the frame energy n db, the mddle graph represents the waveform, the lower surface represents the lp contours. 3.3 Prncpal Component Analyss 15 FP postons were tracked on xy-plane, so each frame s represented by 30 coordnates. In order to mprove effcency of tranng, these hghly redundant vectors are compressed nto 6 weght parameters (w 1 w 6 ) usng PCA: w T = p ( x- x ) ; = ref (1)

4 Where x s the coordnate vector of the actual frame, x ref s the vector of reference frame when the speaker was slent wth closed lps, and p s are the prncpal component vectors. The weght parameters are used to tran NN. For convenence n mplementaton the prncpal component vector are scaled to get the weghts between -1 and +1. In operaton (after tranng phase) our converson algorthm estmates the coordnates from the weghts suppled by NN. The recovery operaton s: 6 x ˆ = x + w p (2) bas = 1 The compresson n our database causes only 1-3% loss of data, whch s 1-2 pxel error n xy-coordnates whch s acceptable n lp-readng. Ths operaton needs only 180 multplcatons. PCA s wdely used n speech anmaton systems due to ts orthogonalty feature whch s utlzed also n MPEG-4 Facal Anmaton standard. 3.4 Talkng head We appled Luca talkng head model [10] wth some modfcaton. It uses the anmaton standard of MPEG-4 called facal anmaton parameters (FAP). Snce FAP s a vseme based anmaton method, we have modfed Luca to work drectly wth feature pont coordnates. Drect control s more generc, so the vertex handlng was refned usng dynamcally weghted movng, whch can be used to avod moton conflctng wth anatomc rules. The number of vertexes n Luca s Fgure 7 - The FP postons expressed by the 1 st, 2 nd 3 rd and 4 th prncpal components usng unqualfed lp-speaker Fgure 6 - The FP postons expressed by the 1 st, 2 nd 3 rd and 4 th prncpal components usng professonal lp-speaker PCA s capable for more than dmenson reducton. The PCA vectors contan mportant nformaton about the lp speaker and the recordng. There are prncpal components, whch have recognzable role n vseme dstncton. (Fg. 6) For example the vertcal movement of the jaw-bone s the strongest prncpal component. Also recognzable prncpal component are horzontal tenson ("cheese" mouth at photos) and lp-roundng. These components have vseme dstnctve functons. In ths pont of vew, only the order of the components s mportant. If the speaker s professonal lp speaker, the strongest components are vseme dstnctve. However, f the speaker s an unqualfed speaker, the correctonal components (eg. emotonal) can outrun vseme dstnctve components. As t can be seen n Fgure 7, the second strongest component shows that the speaker's artculaton s not acceptable for tranng. 4. EXPERIMENTS AND RESULTS 4.1 Prelmnary tests The prelmnary tests were useful n the tunng of the system and n the modfcaton of the database e.g. usng professonal nterpreters/lp-speakers. The prelmnary tests have hghlghted also the mportance of nter speech breaks n the system. Low level background nose n audo speech percepton do not cause any problem but even very small lp movements calculated from the background nose can dsturb very much the lp-readng based speech percepton. After the prelmnary test sessons a dscussons were organsed and remarks, comments, questons of deaf test persons were collected and carefully consdered n refnement steps of our system. 4.2 Fnal tests Lp readng the speech only, phonemes could not be dstngushed perfectly, because some of them have dentcal vseme representaton (lke b-p). The natural way of recognton n those cases mght be the estmaton based on context or startng a dalogue to clarfy the ambguty. To avod ths type of nterruptons the measurng text has to have some redundancy. Our text words were randomly selected from very lmted sets. Two dgt numbers, names of months and

5 names of days were used n our test materal, smlarly to the tranng set. Durng the fnal tests the complete head of the speaker was vsble on large screen. The test materal has been composed randomly from three lp-readng stuatons: A. vdeo records of nterpreter/lp-speaker (no voce), B. face anmaton model controlled by 15 FP coordnates of the nterpreter/lp-speaker (no voce), C. face anmaton model controlled by 15 FP coordnates calculated from speech sgnal (no voce). The test subjects were told to answer the questons n a wrtten form. The tests were composed from 70 short vdeo clps. The complete test was takng about 30 mnutes. In case of sgned requests, the vdeo clp was repeated. 18 deaf persons were nvolved n the tests. See Fg 8. Recognsed words 0% 20% 40% 60% 80% 100% 97,10% 54,90% A B C Fgure 8 - The rato of correctly recognsed words. A vdeo records of nterpreter/lp-speaker, B face anmaton model controlled by FPs os speaker, C face model controlled by FP coordnates calculated from speech sgnal 4.3 Dscusson The vsual word recognton even n the case of natural and professonal speaker has about 3 % of errors. The anmated face model controlled by 15 FP parameters followng accurately the FP parameters of nterpreter/lpspeaker s face resulted about 42% of errors. After test dscussons t was clarfed, that the vsble parts of tongue and movement of parts of the face others then the mouth convey addtonal nformaton to help the correct recognton. Probably the face model tself needs further mprovements. The decreasng of correct recognton only by about 7% as a result the complete changng of face model control from natural parameters to calculated parameters seems to be the fundamental result of our system. 5. CONCLUSION 47,90% The experments and results have proved that the complete speech to facal anmaton converson s possble on the level that provdes communcaton ad for deaf persons. Several components of the system have been mplemented on smart moble phones workng n real tme. The rest of the mplementaton on moble phone s rather a techncal queston. Further mprovement of the facal anmaton model and enhancement of the converson process could reduce the vsual recognton error rate to the absolute tolerable 20% value. Remember, that deaf people have no other natural and effcent telephone based communcaton wth the hearng socety. 6. ACKNOWLEDGEMENT The authors would lke to thank the Natonal offce for Research and Technology for supportng the project n the frame of Contract No 472/04. Many thanks to our hearng mpared frends for partcpatng n many-many tests and for ther valuable advces and remarks. REFERENCES [1] D. Anguta, Matrx Back Propagaton - An efcent mplementaton of the BP algorthm" Techncal Report, DIBE - Unversty of Genova, Nov [2] J. Beskow, Talkng Heads, Models and Applcatons for Multmodal Speech Synthess: Doctoral Dssertaton Stockholm, [3] K. H. Cho and J. N. Hwang, Constraned optmzaton for Audo-to Vsual Converson, IEEE Transactons on Sgnal Processng, Vol. 52. No. 6, pp , June [4] P. Cos, A. Fusaro, G. Tsato, LUCIA a New Italan Talkng-Head Based on a Modfed Cohen-Massaro s Labal Coartculaton Model, Proceedngs of Eurospeech 2003, Geneva, Swtzerland, September 1, 2003, Vol. III, pp [5] B. Granström, I. Karlsson, K-E Spens: SYNFACE a project presentaton Proc of Fonetk 2002, TMH-QPSR, 44: [6] R. Guterrez-Osuna, P.K. Kakumanu, A, Esposto, O. N. Garca, A. Bojorquez, J.L Castllo and I. Rudomn, Speechdrven Facal Anmaton wth Realstc Dynamcs IEEE Transactons on Multmeda, Vol. 7. pp , February [7] M. Johansson, M. Blomberg, K. Elenus, L.E.Hoffsten, A. Torberger, Phoneme recognton for the hearng mpared, TMH-QPSR. vol 44 Fonetk pp , [8] D. W. Massaro, Percevng Talkng Faces: From Speech Percepton to a Behavoral Prncple Cambrdge, Mass: MIT Press, [9] D. W. Massaro, D. G. Stork, Speech Recognton and Sensory Integraton, Amercan Scentst, [10] J. Ostermann, "Anmaton of Synthetc Faces n MPEG-4", Computer Anmaton, pp , Phladelpha, Pennsylvana, June 8-10, [11] G. Salv: Truncaton error and dynamcs n very low latency phonetc recognton Proc of ISCA workshop on Non-lnear Speech Processng (2003)