Downloaded from

Transcription

1 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 34 RealTimeCharacterizationofLipMovements UsingWebcam SureshDS,SandeshC Director&HeadofECEDept,CIT,Gubbi,Tumkur,Karnataka. PGStudent,ECEDept,CIT,Gubbi,Tumkur,Karnataka. Abstract:Communicationplaysanimportantroleinthedaytodayactivitiesofhumanbeing.Themain objectiveofthispaperistohelpthepeoplewhoareunabletospeak.visualspeechplaysamajorroleinthe lipreadingforlistenerswithdeafperson.hereweareusinglocalspatiotemporaldescriptorsinorderto identifyorrecognizethewordsorphrasesfromthedisable(dumb)peoplebytheirlipmovements.local binarypatternsextractedfromthelipmovementsareusedtorecognizetheisolatedphrases.experiments weremadeontenspeakerswith5phrasesandobtainedaccuracyofabout75%forspeakerdependentand 65%forspeakerindependent.WhilebeingmadecomparisonwithotherconceptslikeAVletterdatabase, ourmethodoutperformstheotherbyaccuracyof65%.theadvantagesofourmethodarerobustnessand recognitionispossibleinrealtime. Keywords:LBPTop,Webcam,Visualspeech,KNNclassifier,Silentspeechinterface. I.Introduction Inrecentdays,asilentspeechinterfacehasbecominganalternativeinthefieldofspeechprocessing.A voicebasedspeechcommunicationhassufferedfrommanychallengeswhicharisesduetofactthatspeech shouldbeclearlyaudible,itcannotbemasked,includeslackofrobustnessprivacyissuesandexclusionof speechdisabledperson.sothesechallengesmaybeovercomebytheuseofsilentspeechinterface.silent speech interface is a device, where system enabling speech communication takes place, without the necessityofavoicesignalorwhenaudibleacousticsignalisunavailable.inourapproach,lipmovements arecapturedbytheuseofawebcam,whichisplacedinfrontofthelips.manyresearchworkfocusesonly onvisualinformationtoenhancespeechrecognition[1].audioinformationstillplaysamajorrolethanthe visualfeatureorinformation.but,inmostcasesitisverydifficulttoextractinformation.inourmethodwe are concentrating on the lip movement representations for speech recognition solely with visual information. ExtractionofsetofvisualobservationvectorsisthekeyelementonAVSR(audiovisualspeechrecognition) system.geometricfeature,combinedfeatureandappearancefeaturesaremainlyusedforrepresenting visual information. Geometric feature method represents the facial animation parameters such as lip movement,shapeofjawandwidthofthemouth.thesemethodsrequiremoreaccurateandreliablefacial featuredetection,whicharedifficultinpracticeandimpossibleatlowimageresolution. In this paper, we propose an approach for lip reading or lip movements, where humancomputer interactionimprovessignificantlyandunderstandinginnoisyenvironmentalsoimproves. II.LocalSpatiotemporalDescriptorsforVisualInformation Inthispaper,weareconcentratingonLBPTOPinordertoextractthefeaturesfromtheextractedvideo frames.thelocalbinarypattern(lbp)operatorisagrayscalewhichisnotvaried,alwaysremainsthe sametexture,simplestatistic,whichhasshownthebestperformanceintheclassificationofdifferentkinds oftexture.foranindividualpixelinanimageitsrespectivebinarywillbegeneratedandthethresholdswill becomparedwiththeitsneighborhoodvalueofcenterpixelasshowninfigure1(a)[1].

2 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 35 Figure.1.(a)BasicLBPoperator.(b)Circular(8,2)neighborhood. Wheregcdenotesthegreyvalueofcenterpixel{xc,yc}ofthelocalneighborhoodandgpisrelatedtothe greyvaluesofpequallyspacedpixelsonacircleofradiusr.ahistogramiscreatedinordertocollectup the occurrences of various binary patterns. The definition of neighbors can be extended to circular neighborhoodswithanynumberofpixelsasshowninfigure1(b).bythisway,wecancollectlargerscale textureprimitivesormicropatterns,likespots,linesandcorners. Localtexturedescriptorshaveobtainedtremendousattentionintheanalysisoffacialimage becauseof theirrobustnesstochallengesuchasposeandilluminationchanges.inourapproachatemporaltexture recognitionusinglocalbinarypatternswereextractedfromthethreeorthogonalplanes(lbptop).lbp TOPmethodismoreefficientthantheordinaryLBP.InordinaryLBPweareextractinginformationor featuresintwodimension,whereasinlbptopweareextractinginformationinthreedimensioni.e,x,y andt.forlbptop,theradiiinspatialandtemporalaxesx,y,andt,andthenumberofneighboring pointsinthexy,xt,,andytplanescanalsobedifferentandcanbemarkedasrx,ryandrt, PXY,PXT, PYT.TheLBPTOPfeatureisthendenotedasLBPTOPPXY,PXT,PYT,RX,RY,RT.Ifthecoordinatesof thecenterpixelgtc,eare(xc,yc,,tc),thanthecoordinatesoflocalneighborhoodinxyplanegxy,pare givenby(xcrxsin(2p/pxy),yc+rycos(2p/pxy),tc),thecoordinatesoflocalneighborhoodinxtplane gxt,p are given by (xcrxsin(2p/pxt),yc, tc RT cos(2p/pxt)) and the coordinates of local neighborhoodinytplanegyt,p(xc,ycrycos(2p/pyt),,,tc RTsin(2p/PYT)).Sometimes,theradiiin threeaxesarethesameandsodothenumberofneighboringpointsinxy,xt,andytplanes.inthatcase, weuselbptopp,rforabbreviationwherep= =PXY=PXT=PYTandR=RX=RY=RT[1]. Figure. 2.(a)Volumeofutterancesequence.(b)ImageinXYplane(c) ImageinXTplane(d)ImageinTY plane.

3 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 36 Figure2(a)demonstratesthevolumeofutterancesequence.Figure2(b)showsimageintheXYplane. Figure2( (c)isanimageinthextplaneprovidingavisualimpressionofonerowchangingintime,while Figure2( (d)describesthemotionofonecolumnintemporalspace[1].anlbpdescriptionwhencomputed overthewholeutterancesequenceitencodesonlythe occurrencesofthemicropatternswithoutany indication about their locations.. In order to overcome this effect, a representation which consists of dividingthemouthimageintoseveraloverlappingblocksisintroduced.figure.3alsogivessomeexamples ofthelbpimages.thesecond,third,andfourthrowsshowthelbpimageswhicharedrawnusinglbp code of every pixel from XY (secondrow), XT(third row), and YT(fourthrow)planes, respectively, correspondingtomouthimagesinthefirstrow. Figure3..Mouthregionimages(firstrow),LBPXYimages(secondrow),LBPXTimages(thirdrow),and LBPYTimages(lastrow)fromoneutterance[1]. Figure.4.Featuresineachblockvolume.(a)Blockvolumes.(b)LBPfeaturesfromthreeorthogonalplanes. (c)concatenatedfeaturesforoneblockvolumewiththeappearanceandmotion[1]. Figure.5.Mouthmovementrepresentation[1]. Whenapersonuttersacommandphrase,thewordsarepronouncedinorder,forinstance yousee or seeyou.ifwedonotconsiderthetimeorder,thesetwophraseswouldgeneratealmostthesamefeatures.

4 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 37 Toovercomethiseffect,thewholesequenceisnotonlydividedintoblockvolumesaccordingtospatial regionsbutalsointimeorder,asshowninfigure.4(a)shows.the LBPTOPhistogramsineachblock volume are computed and concatenated into a single histogram, as shown in Figure. 4. All features extractedfromeachblockvolumeareconnectedtorepresenttheappearanceandmotionof themouth regionsequence,asshowninfigure.5. Ahistogramofthemouthmovementscanbedefinedasfollows III.MultiresolutionFeaturesandFeatureSelection Multiresolutionfeatureswillprovidemoreaccurateinformationandalsotheanalysisofdynamiceventwill beincreased.byusingthesemultiresolutionfeatures,ithelpsinincreasingthenumberoffeaturesgreatly. Whenthefeaturesfromthedifferentresolutionweremadetobeconnectedinseriesdirectly,thefeature vectorwouldbeverylongandasaresultthecomputationalcomplexitywillbetohigh.itisobviousthatall themultiresolutionfeatureswillnotcontributeequally,soitisnecessarytofindoutfeatureslikewhich location, withwhatresolutionandmoreimportantlythetypessuchasappearance,horizontalmotionor verticalmotionthatareveryimportant.weneedfeatureselectionforthispurpose.inchanging the parameters,threedifferenttypesofspatiotemporalresolutionarepresented:1)useofadifferentnumberof neighboringpointswhencomputingthefeaturesinxy(appearance),xt(horizontalmotion),andyt (verticalmotion)slices;2)useofdifferentradiithatcancapturetheoccurrencesindifferentspaceand timescales;3)useofblocksofdifferentsizestocreateglobalandlocalstatisticalfeatures[4][5]. IV.OurSystem Oursystemconsistsofthreestages,asshowninFigure6.Thefirststageisadetectionlipmovements.The secondstageextractsthevisualfeaturesfromthemouthmovementsequence.theroleofthefinalstageis torecognizetheinpututteranceusinganknnclassifier. Figure6.Systemdiagram

5 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 38 Inourapproachwebcamisusedtocapturethelipmovements.Furtherwewillextractthevisualfeature fromthe mouth region. These features are thenfedto the classifier. For speech recognition, aknn classifierisselectedsinceitiswellfoundedinstatisticallearningtheoryandhasbeensuccessfullyapplied tovariousobjectdetectiontasksincomputervision.sincetheknnisonlyusedforseparatingtwosetsof points,thephraseclassificationproblemisdecomposedintotwoclassproblems,thenavotingschemeis usedtoaccomplishrecognition.sometimesmorethanoneclassgetsthehighestnumberofvotes;inthis case,1nntemplatematchingisappliedto theseclassestoreachthefinalresult.thismeansthatin training,thespatiotemporallbphistogramsofutterancesequencesbelongingtoagivenclassareaveraged togenerateahistogramtemplateforthatclass[1]. V.ExperimentsProtocolandResults Formoreaccurateevaluationofourproposedmethod, wemadeadesignwithdifferentexperiments, includingspeakerindependent,speakerdependent,multiresolution. 1.SpeakerIndependentExperiments:Forthespeakerindependentexperiments,leaveonespeakeroutis utilized.wemadeatrainingofparticularphraseusingonespeakerandwemadetosaythesamephrase with10differentspeakersandwhentestingwasdoneweweresuccessfulingettingthesamephrasefrom6 speakers.theoverallresultswereobtainedusingm/n( (Misthetotalnumberofcorrectlyrecognized sequencesandnisthetotalnumberoftestingsequences) ).Whenweareextractingthelocalpatterns,we takeintoaccountnotonlylocationsofmicropatternsbutalsothetimeorderoflipmovements,sothe wholesequenceisdividedintoblockvolumesaccordingtonotonlyspatialregionsbutalsotimeorder. Figure.7demonstratetheperformanceforeveryspeaker.Theresultsfromthesecondspeakerarethe worst,mainlybecausethebigmoustacheofthatspeakerreallyinfluencestheappearanceandmotioninthe mouthregion. Figure.7.Recognitionperformanceforeveryspeaker 2)SpeakerDependentExperiments:Forspeakerdependentexperiments,theleaveoneutteranceoutis utilizedforcrossvalidation.inourapproachwhenaparticularspeakeristrainedwithsomelistofphrases andaftertestingisperformedwiththesamespeaker70%ofsamephraseswereidentifiedcorrectly. 3)OneOneversusOneRestRecognition:Inthepreviousexperimentsonourowndataset,thetenphrase classificationproblemisdecomposedinto45twoclassproblems( Hello Seeyou, Iamsorry Thank you, Youarewelcome Haveagoodtime, etc.).butusingthismultipletwoclassstrategy,thenumber ofclassifiersgrowsquadraticallywiththenumberofclassestoberecognizedlikeinavlettersdatabase. WhentheclassnumberisN,thenumberoftheKNNclassifierswouldbeN(N1)/2. Figure.8.Selected15slicesforphrases Seeyou and Thankyou. intheblocksmeanstheytslice (verticalmotion)isselected,and _ thextslice(horizontalmotion), / meansthe appearancexyslice[1].

6 Proceedings of The National Conference on Man Machine Interaction 2014 [NCMMI 2014] 39 Figure8showstheselectedslicesforsimilarphrases seeyou and thankyou.thesephraseswerethe mostdifficulttorecognizebecausetheyarequitesimilarinthelatterpartcontainingthesameword you. Theselectedslicesaremainlyinthefirstandsecondpartofthephrase;justoneverticalsliceisfromthe lastpart..theselectedfeaturesareconsistentwiththehumanintuition. TableI ResultsfromOnetoOneandOnetoRestClassifiersonSemiSpeakerDependentExperiments (Resultsin TheParenthesesareFromOnetoRestStrategy) Conclusion Arealtimecaptureoflipmovementsusingwebcamforrecognitionofspeechwasproposed, inorderto helpthedisableperson.ourapproachuseslocalspatiotemporaldescriptorsfortherecognitionofinput utterance.lbptopisusedtoextractthefeaturesfromthecapturedimages.experimentsweremadeon ten speakers and the lip movements were converted into speech very efficiently. Compared to other approach,ourmethodoutperformstheotherbyaccuracyof70%. References 1. LipreadingWithLocalSpatiotemporalDescriptorsIEEETransactionsOnMultimedia,Vol.11,No.7, November T.Ahonen,A.Hadid,andM.Pietikäinen, Facedescriptionwithlocalbinarypatterns:Application tofacerecognition, IEEETrans.PatternAnal.Mach.Intell.,vol.28,no.12,pp , G.ZhaoandM.Pietikäinen, Dynamictexturerecognitionusinglocalbinarypatternswithan applicationtofacialexpressions, IEEETrans.PatternAnal.Mach.Intell.,vol.29,no.6,pp , G.Zhao,M.Pietikäinen,andA.Hadid, Localspatiotemporaldescriptorsforvisualrecognitionof spokenphrases, inproc. 2ndInt.WorkshopHumanCenteredMultimedia(HCM2007) ),2007,pp G. Zhao and M. Pietikäinen, Boosted multiresolution spatiotemporal descriptors for facial expression recognition, Pattern Recognit. Lett.,, Special Issue on Image/VideoBased Pattern AnalysisandHCI,2009,acceptedforpublication.