PHONETIC CONSEQUENCES OF SPEECH DISFLUENCY

Transcription

1 PHONETIC CONSEQUENCES OF SPEECH DISFLUENCY Elizabth E. Shribrg Spch Tchnology and Rsarch Laboratory SRI Intrnational, Mnlo Park, CA ABSTRACT Unlik rad or laboratory spch, spontanous spch contains high rats of disfluncis (.g., rptitions, rpairs, filld pauss). Such vnts rflct production problms frquntly ncountrd in vryday convrsation. Analyss of Amrican English show that disfluncy affcts a varity of phontic aspcts of spch, including sgmnt durations, intonation, voic quality, vowl quality, and coarticulation pattrns. Ths ffcts provid clus about production procsss, and can guid mthods for disfluncy procssing in spch rcognition applications. 1. INTRODUCTION A clar diffrnc btwn spontanous spch and rad or laboratory spch is that th formr contains significant rats of disfluncis (.g., filld pauss, rptitions, and rpairs), such as Filld paus uh, um Rptition th th Rpair any halth cov- any halth insuranc Fals Start It s fir- I could gt it whr I work In laboratory or rad spch, whr contnt is givn or highly constraind, minimal complx procssing is rquird. But in vryday convrsation, our mssags ar constructd on th fly. W must dcid what to say, how to say it, and how to coordinat our intractions with othrs all in ral tim. It is thus hardly surprising that w somtims nd to paus, or to rpair our prvious spch. Rats of disfluncy pr word in spontanous English spch vary from undr 1% for constraind human-computr dialog, to 5-10% for natural convrsations [12, 16, 4, 19]. Thr is also considrabl variation across spaking nvironmnts in th rlativ rats of particular disfluncy typs [12, 19]. Such distributional diffrncs rflct diffrncs in cognitiv dmands, and in managing intraction in convrsation [9, 2]. Whil considrabl past work has focusd on lxical proprtis of disfluncy, rcnt yars hav sn incrasing attntion to th qustion of phontic proprtis. An arly suggstion by Hindl [7] proposd that disfluncis ar markd by a spcial acoustic dit signal at intrruption. Although inspction [1], as wll as psycholinguistic xprimnts [11], has rvald no such spcific signal, disfluncy is nvrthlss associatd with a varity of phontic charactristics that diffrntiat it from flunt spch. Th goal of this papr is to outlin som of th main phontic consquncs of disfluncy. As w will s, such ffcts can provid a window onto production procsssthat a lxical or wordlvl analysis oftn obscurs. Thy can also guid dvlopmnt of improvd modls for disfluncy procssing in spch applications. 2. THE STRUCTURE OF DISFLUENCIES Th majority of disfluncis that occur in spontanous spch can b analyzd as having th following thr-rgion surfac structur (trms adaptd from Lvlt [8]): (Prior Rpar- Editing Rpair (Continuation) contxt) andum Phas. um w r fin it s. uh aftr fiv hav th. th tools to rs-. rlax at hom all this. this papr it s. did you? Th first rgion of th disfluncy is th rparandum,or matrial that will latr b rplacd. Th nd of this rgion corrsponds to th intrruption point (markd with a. ) or th location at which thr is a dpartur from fluncy. By this point, th spakr has dtctd som problm, and according to a Main Intrruption Rul halts th production procss [8].) Th diting phas consists of th rgion from th intrruption point to th onst of th rpair. This rgion may b mpty, contain a silnt paus, or contain diting phrass or filld pauss ( I man, um, uh ). Th trm diting is not intndd to imply dtction of rror; pausing can occur for rasons not involving rror. Finally, w hav th rpair rgion, which typically rflcts th rsumption of fluncy. (W will assum hr that th rpair is not itslf followd by anothr slf-intrruption. If it is, th disfluncy is complx [19].) Ths rgions ar contiguous, andrmoval of th first two (rparandumandditing phas) yilds a lxically flunt vrsion. As shown, w can analyz all of our disfluncy typs this way. A disfluncy may contain matrial only in th diting phas, such as a filld paus. Or it may contain only rpatd words in th rparandum and rpair. Not that for rpats such as th th, this structur prdicts that it is th first instanc, and not th rpatd on, that is most likly to b abrrant, a prdiction w will s latr vidnc for basd on phontic faturs. Editing trms can combin with diffrnt typs of disfluncy (.g., th uh th ; rsi man rlax ). W will organiz our ovrviw of phontic consquncs by moving through ths thr rgions lft to right, discussing th ffcts in ach. As w will s, most of th proprtis ar in th rparandumand diting phas, but crtain ffcts can also b sn in th rpair. 3. EFFECTS IN THE REPARANDUM Although at a lxical lvl of rprsntation th rparandum is rmovd in full to arriv at a flunt lxical vrsion, it is not until

2 Rport Documntation Pag Form Approvd OMB No Public rporting burdn for th collction of information is stimatd to avrag 1 hour pr rspons, including th tim for rviwing instructions, sarching xisting data sourcs, gathring and maintaining th data ndd, and complting and rviwing th collction of information. Snd commnts rgarding this burdn stimat or any othr aspct of this collction of information, including suggstions for rducing this burdn, to Washington Hadquartrs Srvics, Dirctorat for Information Oprations and Rports, 1215 Jffrson Davis Highway, Suit 1204, Arlington VA Rspondnts should b awar that notwithstanding any othr provision of law, no prson shall b subjct to a pnalty for failing to comply with a collction of information if it dos not display a currntly valid OMB control numbr. 1. REPORT DATE REPORT TYPE 3. DATES COVERED to TITLE AND SUBTITLE Phontic Consquncs of Spch Disfluncy 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) SRI Intrnational,Spch Tchnology and Rsarch Laboratory,Mnlo Park,CA, PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approvd for public rlas; distribution unlimitd 13. SUPPLEMENTARY NOTES 14. ABSTRACT s rport 15. SUBJECT TERMS 11. SPONSOR/MONITOR S REPORT NUMBER(S) 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT b. ABSTRACT c. THIS PAGE 18. NUMBER OF PAGES 4 19a. NAME OF RESPONSIBLE PERSON Standard Form 298 (Rv. 8-98) Prscribd by ANSI Std Z39-18

3 th spakr notics troubl that w should xpct to s phontic manifstations. Indd this is what w find. Phontic ffcts in th rparanda of disfluncis ar most prvalnt at or around th intrruption point Duration Pattrns On of th most prvasiv ffcts of disfluncy is a lngthning of rhyms or syllabls immdiatly prcding th intrruption point. Effcts can at tims b sn to xtnd furthr back, into full words or phrass. As an xampl w will look at on-word rptitions such as th th in th Switchboard corpus of human-human tlphon convrsations [6]. To assss lngthning, w compar th durations of ach instanc to th duration of th in flunt contxts. Rsults ar shown in Figur 1; thy rprsnt data from a singl spakr. As can b sn, th rparandum (Rp1) is lngth- Man Duration (millisconds) Rp1 Rp2 Unrpatd th th th th th SWBD Figur 1. Duration of Words in Rptitions and Flunt Countrparts. R1=1st instanc (rparandum), R2=2nd instanc (rpair). nd considrably rlativ to its xpctd duration in flunt spch, whras th rpair (Rp2) has about th sam duration as th flunt countrpart. This suggsts that in rptitions, spakrs ar drawing out th rparandum much lik thy might a filld paus. Howvr not all rptitions show this pattrn. A mor dtaild study shows that thr ar at last thr main typs of rpats whn classifid basd on prosodic proprtis, suggsting at last thr diffrnt undrlying stats of th spakr whn rpating [17]. Th pattrn dpictd in Figur 1 howvr corrsponds to th most common cas. Durational lngthning in th rparandum is sn for othr disfluncy typs as wll, and is on way spakrs can paus without casing phonation [3] Intonation Intrstingly, whn spakrs modify duration, thy tnd to do so in a way that prsrvs intonation pattrns and local pitch rang rlationships. For xampl whil th rparandum in a rptition is oftn xtndd in duration, it typically shows a pitch contour similar to that ofits following countrpartin th rptition, but just strtchd out ovr mor tim as shown in Figur 2 (pitch tracks ar indicatd for only th words in th rptition): 3.3. Word Cutoffs and Laryngalization In rad or laboratory spch, w xpct words to b compltd, but this is not th cas in spontanousspch. Spakrs halt production soon aftr noticing troubl [8], without concrn for word boundaris. In a corpus of human-computr dialog on air travl planning (ATIS; [13]) narly 60% of disfluncis containd word cutoffs; rats in two human-human corpora wr about 20-25% [19]. Th diffrnc is largly du to th highr rlativ rat of rror rpairs in human-computr dialog. Errors ar not highr ovrall in such corpora, but bcausnon-rror hsitations (filld pauss and rptitions) ar supprssdin human-computrdialog with a push-to-talk mchanism for spch input, rrors mak up a largr proportion of total disfluncis. Various rsarchrs hav dscribd cutoffs as abrupt, showing som form of laryngalization [1, 14, 11]. In an analysis of cutoffs in th ATIS data conductd by Madlain Plauché, w found that a typical form of laryngalization in such casscorrspondsto craky voic on th last ms of th cut off words. Howvr, it is not th cas that all cutoffs ar suddn, or that word cutoffs always corrspond to rrors. On th contrary, th highst rat of cutoffs foundin th ATIS corpus wasonsimpl rptitions. Hr th rat was ovr 70% of rpats, whras rats for rpairs of rror wr undr 50%. And som cutoffs could b xtndd in duration, mor indicativ of hsitation than of suddn dtction of rror. Cut off words prsnt a problm for automatic spch rcognition sinc partial-word pronunciations ar not prsnt in th dictionary. Although on could add all possibl initial phon squncs of a word as possibl pronunciations, such an approach would crat a prolifration of pronunciations that would only hurt prformanc by incrasing confusability. A possibl solution is to constrain fragmnts to b rcognizd only as parts of closly following words Coarticulation Anothr consqunc of disfluncy is a chang in surfac coarticulation pattrns. In th production of words in flunt spch, articulators gnrally mov toward th articulator positions for th onst of th nxt word. But in disfluncis, this proximal rlationship of coarticulation to actual output word squnc cannot b assumd. Coarticulation is govrnd by th nxt word in th spakr s phontic plan at th tim th word in qustion is uttrd not by th word squnc that is ultimatly producd. In flunt spch, th plan and th final output ar consistnt, but in disfluncis, following lxical contnt may b tmporarily unavailabl, or th plan can chang on th fly. W lookd at this qustion in a study of singl-word rpats of th and I. Not that only th plac of articulation can safly b dtrmind for transitions, although thr ar som cass whr th mannr is clar. W will classify cass basd on consistncy with a surfac word, although of cours w cannot know for sur whthr som othr word was intndd. Blow ar rsults with illustrativ xampls; transitions ar markd in parnthss, using standard orthography. Transition Frquncy Exampl (a) NONE 722 (88%) th. th dog (b) to word aftr rpat 71 (9%) th(d). th dog (c) to diffrnt word 19 (2%) th(d). th cat (d) to rpat itslf 3 (.3%) th(th). th dog Figur 2. Pitch of Rpatd Words. As shown, most cass of rpats hav no dtctabl final transition. This is diffrnt from what is xpctd in flunt connctd spch; hr most cass containd a paus at intrruption. For

4 spch rcognition modls, w may thus want to turn off crossword modling at rptition boundaris, or mor gnrally at th intrruption point of disfluncis. Th nxt two cass ar also quit intrsting, bcaus thy show coarticulation that is inconsistnt with th following surfac word. Cas (b), which rprsnts th majority of cass with coarticulation, shows that somtims disfluncy ffcts can b sn arlir than th location of th lmnt causing troubl. From th transition w can infr that th spakr committd to th word dirctly aftr th rptition but stalls arlir, prhaps to kp syntactic or prosodic units intact. Cas (c) is almost crtainly a covrt rpair, whr som word othr than cat was caught bfor it was uttrd, and rpaird. Cas (d) is standard in trms of having a transition consistnt with th actual following word, but notic that th following word is th rpat itslf. This suggsts that in som cass, spakrs must b planning to rpat whil thy ar still producing th first instanc of th word. As with cas (a), cass (b) and (c) also pos problms for cross-word modling in spch rcognition. This tim, th problm is that thr is acoustic vidnc for a sgmnt at th nd of th rparandum that is inconsistnt with rcognizr modls constraind to modl pronunciation only across contiguous surfac words Vowl Quality Disfluncy is also associatd with altrations in vowl quality. A spcialcas is th word th, which has an altrnat pronunciation, [dh iy], bfor vowl-initial words in many dialcts of Amrican English. This altrnat is also mor likly in th rparandum of rptitions [5]. Othr words without suchvariants, but with citation forms that diffr from thir pronunciation in connctd spch, show a similar bhavior (although it is not clar whthr thy rflct th sam phnomnon). For xampl, a and to ar much mor likly to b pronouncd as [y] and [t uw] in th rparandum of disfluncis than lswhr. It is not clar whthr such forms ar producd as signals to listnrs, or whthr thy rflct a modification rlatd to othr acoustic proprtis such as durational lngthning and following pauss. Howvr it is clar that spakrs choos th altrnat form bfor uttring th word, bcaus vowl quality nvr shifts within th word itslf. 4. EFFECTS IN THE EDITING PHASE 4.1. Unfilld Pauss Undr Lvlt s framwork of spch production [9], slfintrruption is associatd with a halting of th spch production procss at all lvls. Thrfor, som minimum tim is ndd aftr th spch is cut off in ordr to plan th rpair. Disfluncy is thus oftn indicatd by unfilld pauss in th diting phas. For automatic spch procssing of disfluncis, ths pauss hav provn to b vry usful. Work using dcision trs to modl acoustic faturs finds that pauss ar among th bst cus to disfluncy dtction [20, 21], bcaus thy ar robustly xtractd and nsur high rcall Filld Paus Duration In English, th vowl in th filld pauss um and uh is typically clos to schwa; howvr, it can also carry strss, or occur furthr back and lowr in th vowl spac. In automatic spch rcognition, Filld pauss ar somtims misrcognizd as a or as parts of othr words containing th rlvant vowls. But filld pauss diffr dramatically from ths othr instancs in duration. To illustrat, durations for th vocalic portion of 700 filld pauss and for 40,000 instancs of th sam vowls lswhr, including in th dtrminr a, wr obtaind from rcognizr forcd alignmnts using th ATIS corpus. Rsults ar shown in Figur 3. Normalizd Frquncy [ ],[a],[^] in filld paus [ ] in dtrminr a [^] lswhr [ ], [a] lswhr ATIS Duration (sconds) Figur 3. Duration of Vowls in Filld Pauss and Elswhr. As shown, vowls in filld pauss hav much longr durations than th sam vowls in flunt contxts. Duration, thn, is a simpl cu that could b usd by spch rcognition systms in discriminating vowls in filld pauss from th sam vowls lswhr. It is also important to trat such durations sparatly in duration modling for othr purposs, so as not to skw th distributions for ths vowls Filld Paus Intonation Filld pauss hav bn shown to b low in F0, and display a gradual, roughly linar pitch or fundamntal frquncy (F0) fall [15]. In addition, th F0 of filld pauss occurring within a claus was found to b rlatd to th F0 of th surrounding spch [18]. Figur 4 shows F0 valus for th onst and offst of a filld paus, and th prcding and following F0 paks. Lins connct points for a spcific filld paus. Th four F0 masurmnts ar plottd at qually spacd intrvals; thrfor th actual tmporal intrvals btwn ths points (which varid gratly) ar not rprsntd in th figur. Th solid havy lin indicats th spakr s stimatd baslin F0, as stimatd by masuring F0 at th nd of sntncfinal F0 falls. Fundamntal Frquncy (Hz) ATIS fmal spakr baslin Prcding Pak Onst of FP Offst of FP Following Pak Figur 4. F0 of Filld Pauss and Surrounding Paks. What is striking hr is that th F0 of filld pauss falls about halfway btwn th prcding pak valu and th spakr baslin. In fact, F0 valus in th study wr wll prdictd by a simpl additiv-multiplicativ modl basd on ths valus. Ths rlationships hld dspit considrabl diffrncs in tim intrvals btwn th four masurd valus plottd at rgular intrvals as in Figur 4. Ths findings suggst that for filld pauss, similar to what w saw arlir for rptitions, spakrs may prsrv intonational rlationships undr changs in duration ncssitatd by th nd to paus.

5 5. EFFECTS IN THE REPAIR As said arlir, most consquncs of disfluncy ar locatd in th rparandum and diting phas, sinc th rpair rgion constituts th onst of fluncy. An xcption, howvr, is that crtain typs of rpair can show ffcts of having mad a chang in contnt, in th form of contrastiv mphasis on th rpairing lmnt. Lvlt and Cutlr [10] lookd at prosodic marking, or an incras in F0, duration, or amplitud, in th rpair rgion of disfluncis from a pattrn dscription task. Thy found that marking occurrd for roughly half of th rpairs involving rror, and for only about 20% of th rpairs involving mr laboration. This suggsts that it may b mor important to call attntion to outright rror than to inappropriatnss. Such marking also illustrats that w cannot simply rmov th rparandum and diting phas, laving a prfctly flunt rpair. All thr rgions ar still in th discours rcord; th prosodic contrast in th rpair is producd with rspct to th arlir mntion in th rparandum. 6. SUMMARY AND CONCLUSION Disfluncis ar rar in laboratory spch, but occur with considrabl frquncy in vryday communication. Most disfluncis can b analyzd as having a thr-rgion structur, in which th first two rgions ar rmovd to yild a flunt vrsion of th uttranc. Disfluncy affcts a varity of phontic aspcts of spch, mainly in th two rgions that ar rmovd. Th ffcts includ changs in sgmnt durations, intonation, word compltion, voic quality, vowl quality, and coarticulation pattrns. Ths ffcts provid insights into th mchanisms undrlying th production of spontanous spch in conditions charactristic of th ral world. In addition thy provid information that can b usd to bttr modl disfluncisin automatic spchrcognition applications. ACKNOWLEDGMENTS I am indbtd to Madlain Plauché for xtnsiv acoustic analyss and hlpful discussion. Andras Stolck, Dilk Hakkani-Tür, and Gökhan Tür providd assistanc with softwar and data procssing. This rsarch was supportd by NSF and DARPA, undr NSF Grant IRI and NSF- STIMULATE IRI Th viws ar thos of th author and should not b intrprtd as rprsnting thos of th funding agncis. REFERENCES [1] Bar, J., Dowding, J., and Shribrg, E. E Intgrating multipl knowldg sourcs for dtction and corrction of rpairs in humancomputr dialog. In Procdings of th 30th Annual Mting of th Association for Computational Linguistics, Univrsity of Dlawar, Nwark, Dlawar. [2] Clark, H Using Languag. Cambridg: Cambridg Univrsity Prss. [3] Duz, D Acoustic corrlats of subjctiv pauss. Journal of Psychological Rsarch, 22(1), [4] Fox Tr, J. E Th ffcts of fals starts and rptitions on th procssing of subsqunt words in spontanous spch. Journal of Mmory and Languag, 34, [5] Fox Tr, J. E. and Clark, H. H Pronouncing th as th to signal problms in spaking. Cognition, 62, [6] Godfry, J., Holliman, E., and McDanil, J SWITCHBOARD: Tlphon spch corpus for rsarch and dvlopmnt. In Procdings of th IEEE Confrnc on Acoustics, Spch, and Signal Procssing, vol. 1, pp [7] Hindl, D Dtrministic parsing of syntactic non-fluncis. In Procdings of th 21st Annual Mting of th Association for Computational Linguistics, pp [8] Lvlt, W. J. M Monitoring and slf-rpair in spch. Cognition, 14, [9] Lvlt, W. J. M Spaking: From Intntion to Articulation. Cambridg, MA: MIT Prss. [10] Lvlt, W. J. M. and Cutlr, A Prosodic marking in spch rpair. Journal of Smantics, 2(2), [11] Lickly, R. J Dtcting Disfluncy in Spontanous Spch. Ph.D. thsis, Univrsity of Edinburgh. [12] Maclay, H. and Osgood, C. E Hsitation phnomna in spontanous English spch. Word, 15, [13] MADCOW Multi-sit data collction for a spokn languag corpus. In Procdings of th 5th DARPA Spch and Natural Languag Workshop. Morgan Kaufmann. [14] Nakatani, C. H. and Hirschbrg, J A corpus-basd study of rpair cus in spontanous spch. Journal Acoustical Socity of Amrica, 95(3), [15] O Shaughnssy, D Rcognition of hsitations in spontanous spch. Procdings of th IEEE Confrnc on Acoustics, Spch, and Signal Procssing, I, [16] Oviatt, S. L Prdicting spokn disfluncis during humancomputr intraction. Computr Spch and Languag, 9, [17] Plauché, M. and Shribrg, E Data-drivn subclassification of disflunt rptitions basd on prosodic faturs. In Procdings of th Intrnational Congrss of Phontic Scincs, San Francisco. [18] Shribrg, E. E. and Lickly, R. J Intonation of claus-intrnal filld pauss. Phontica, 50, [19] Shribrg, E Prliminaris to a Thory of Spch Disfluncis. Ph.D. thsis, Univrsity of California, Brkly, CA. [20] Shribrg, E., Bats, R., and Stolck, A A prosody-only dcision-tr modl for disfluncy dtction. In Procdings of th 5th EuropanConfrncon SpchCommunication and Tchnology, vol. 5, pp [21] Stolck, A., Shribrg, E., Bats, R., Ostndorf, M., Hakkani, D., Plauché, M., Tür, G., and Lu, Y Automatic dtction of sntnc boundaris and disfluncis basd on rcognizd words. In Procdings of th Intrnational Confrnc on Spokn Languag Procssing, vol. 5, pp