Pitch Estimation Enhancement Employing Neural Network- Based Music Prediction

Transcription

1 Ptch Estmato Ehacemet Employg eural etwor- Based Musc Predcto Mare Szczerba & Adrze Czyżews Soud & Vso Egeerg Departmet Techcal Uversty of Gdańs ul. arutowcza /, PL-895 Gdańs, Polad tel. +48 (58) mare@austya.com ABSTRACT I ths paper a ew method for ptch estmato ehacemet was preseted. Ptch estmato methods are wdely used for extractg muscal data from dgtal sgal. A bref revew of these methods s cluded the paper. However, sce processed sgal may cota ose ad dstortos, the estmato results ca be erroeous. The proposed method was developed order to overrde dsadvatages of stadard ptch estmato algorthms. The ew -approach s based o both ptch estmato terms of sgal processg ad ptch predcto based o muscal owledge modelg. Frst, sgal s parttoed to segmets roughly aalogous to cosecutve otes. Thereafter, for each segmet a autocorrelato fucto s calculated. Autocorrelato fucto values are the altered usg ptch predctor output. A musc predctor based o artfcal eural etwors was troduced for ths tas. The descrpto of the proposed ptch estmato ehacemet method s cluded ad some detals cocerg musc predcto are dscussed the paper.. ITRODUCTIO Ptch estmato s oe of the mostly vestgated ad developed areas of sgal processg [][4]. Ptch estmato methods are wdely used for musc trascrpto acqusto of muscal data from dgtal sgal ad for musc strumet tmbre parameterzato [7]. These methods ad ther musc trascrpto performace are brefly revewed the paper. However, ptch estmato methods appled for automatc musc trascrpto from acoustc sgal cause umerous processg errors. There are two ma types of such errors: traset errors ad octave errors []. Coversely, may of such cases huma lsteers ca determe ptch of sgal evdetly. Above remars state the motvato for the preseted wor. Revsg psychophysologc costrats [3] two ma resultat assumptos were troduced: huma lsteers tegrate ptch wth the durato of sgular ote (.e. betwee trasets) ad they ca predct ptch of cosecutve otes, so they are able to correct musc formato spte of osy or dstorted sgal. These assumptos are fudametal for the ew ptch estmato method. Sgal s processed wth segmets roughly equvalet to cosecutve otes (ptch tegrato) ad the predcted for each ote (ptch predcto).. PITCH ESTIMATIO METHODS AD THEIR PERFORMACE There are umerous methods of ptch estmato developed by a umber of researchers. These methods are maly categorzed terms of fuctoal doma: there are tme, frequecy, tme-frequecy ad cepstrum methods [4]. Ptch estmate ca be evaluated tme-doma by detfyg perodcty features wth the soud wave. The most commoly used tme-doma ptch estmato methods clude: threshold-crossg aalyss methods, parallel processg method, evelope aalyss, autocorrelato ad AMDF methods. Frequecy-doma ptch estmato ca be evaluated by detfyg certa features wth the short-term spectra of muscal sgal. Bloc dagram of a geeral frequecydoma ptch estmator s show Fgure. pre-emphass parttog (frames) wdow fucto w S a x f zeropaddg FFT perodcty feature aalyss Fgure. Geeral frequecy-doma ptch estmator dagram. Frequecy-doma ptch estmators clude the followg: Schroeder s hstogram ad spectral compresso methods, comb-flter method ad Beauchamp s method. Ptch ca be also estmated usg some tme-frequecy methods as sub-bad processg based o Medds-Hewtt model [6] ad McAulay-Quater method. Aother, yet popular method s estmato of ptch cepstral doma. All of the above ptch estmato methods were mplemeted ad ther performace has bee tested usg umerous sgals. The sgals were geerated usg commo wavetable sytheszer (Soud Blaster PCI card). The excerpts from real recordgs (oboe ad vol solo) were also used for the expermets. All fles were moophoc, sampled usg.5 Hz sample rate ad 6- bt resoluto. P

2 Ptch estmator s performace was evaluated usg the followg measures: e = f ˆ f = where: ˆf s the estmated ad f - the real fudametal frequecy, df pr = r = where: dla f < fˆ < f r = (3) ˆ dla f < f fˆ > f gvg the percetage of correctly estmated fudametal frequeces wth semtoe precso.. Expermets A set of sytheszed sgals was prepared for the expermets. These clude sgular otes as well as musc phrase usg sytheszed tmbres of flute, oboe, orga ad pao. Ptch estmato methods performace was also verfed usg real musc recordgs. Two excerpts were used for the expermets: a excerpt from Fatasa o. from Fatases for Oboe Solo by G. Ph. Telema [4] ad a excerpt from Caprcco A-mor o. 4 from 4 Caprccos for Vol by. Paga []. Expermets were performed usg the followg ptch estmato methods: autocorrelato, comb-flter, cepstral ad Medds-Hewtt model. Performace was evaluated usg e ad df pr measures relatvely to the geerated referece dataset phrase defto based o musc otato, verfed by a expert. Expermets cocerg the autocorrelato method cofrm that weghtg ad atteuato sgfcatly mprove ptch estmato performace. The hghest performace ratg (df pr ) for the weghted ad atteuated autocorrelato was aroud 86% whereas for stadard method 68%. Comb-flter method was evaluated for a varable fadg factor. Performace ratg was slghtly lower tha for the weghted ad atteuated autocorrelato method. However, sce obtaed optmal fadg factor was varable, the comb-flter method seems to be more uversal ad ot to requre adaptato accordgly to sgal s characterstcs. Revsg cepstral ptch estmator s performace based o the tests wth the sytheszed sgals a addtoal cepstrum atteuator was troduced. The atteuated cepstrum s defed as: CM = log ( + ) C (4) where C deotes the cepstrum ad s a atteuato rate. The accuracy ratg acheved for the cepstral ptch estmator was lower tha for the autocorrelato ad combflter methods aroud 75%. Tests for the Medds-Hewtt model-based ptch estmator proved t s hgh accuracy ad comprehesveess. A absolute ratg was slghtly lower tha for autocorrelato method (aroud 83%), though Medds-Hewtt model-based estmator does ot requre adaptato for dvdual sgal s characterstcs as case of the other methods. It should be oted however, that sce Medds-Hewtt model-based ptch estmator performs complex sgal aalyss sub-bads t requres hgh computg effort. For all the methods mplemeted ptch estmato accuraces were examed. There are two ma error categores: errors caused by traset oses ad dstortos occurrg betwee otes (traset errors) ad errors orgated from temporal harmoc structure cossteces causg octave shfts (octave errors). As oted the Itroducto two ma psychoacoustc merts for the ptch estmato: ptch tegrato through tme ad ptch predcto were employed to reduce ptch estmato errors. The detals cocerg troduced ptch estmato support method are preseted the further sectos of ths paper. 3. PITCH ITEGRATIO Based o the evaluato of several ptch estmato algorthms preseted above a ew method corporatg ptch tegrato through tme was elaborated. The autocorrelato method was selected as a fudametal processg route for ptch estmato. 3. Sgal Segmetato The am of the sgal segmetato method s to dvde a moophoc musc sgal to segmets roughly correspodg to dvdual otes wth a excerpt. Covetoal methods perform segmetato upo ampltude evelope aalyss. Such soluto ca be effcet case of sgals cosstg of clearly separated subsequet otes. However case of real musc recordgs cludg legato artculato ad sgfcat reverberato a alterate soluto should be formulated. It has bee observed, that durg traset portos of sgal maxmum values of autocorrelato fucto ca decrease sgfcatly. Also sgal ampltude ca drop off mometarly. Ampltude varato ca be aalyzed terms of a followg fucto: M log x( m) (5) M m= where M s the perod of the lowest prospectve fudametal frequecy perod. Cosequetly a partto fucto s g was commeced as follows: M max ρ + log x( h + m) s = (6) g wm m= where ρ s a weghed ad atteuated autocorrelato fucto ad w s a scalg factor (accordg to the tal expermets scalg factor was set to w = ) ad h s a leap sze of the aalyss. Trasets locato wth a sgal ca be estmated upo the posto of mmums of the s g fucto. However, case of legato artculato ad sgfcat reverberato sgal eergy as well as maxmum value of the autocorrelato fucto may ot fade out wth traset. Therefore, the segmetato route based o

3 s g fucto may ot wor correctly. Durg steady sectos of a sgal, local autocorrelato peas do ot alter cosderably. However wth trasets some autocorrelato peas fade away whereas the ew oes appear. Cosequetly, a secod segmetato fucto s h was troduced: s h L ( ) = ( lmax ) ρ ( lmax ) l= ρ (7) where l max deotes l-th terms of ampltude local maxmum of the autocorrelato fucto ρ. Both segmetato fuctos ca be cocered complemetary. Therefore, basg o tal expermets a followg comprehesve segmetato fucto was establshed: s = s qs (8) [ ] f g + h where q s a scalg factor set tetatvely to q = 4. A llustrato of the segmetato fucto s f s preseted Fgure. s f Fgure. s f fucto plot. Accordgly, segmetato may be executed upo locato of mma wth s f fucto. It ca be assumed that trasets caot occur desely through tme. Therefore segmetato pots postoed dstace less tha samples from the prevous oes are elmated. Ital expermets proved the method s capablty to partto a muscal excerpt to segmets roughly equvalet to dvdual subsequet otes. It has also bee observed that the troduced segmetato method may wor correctly case of glssado-le ptch alteratos. 3. Ptch Estmato wth Segmets It ca be assumed, that ptch may alter sgfcatly wth a desgated segmet. Cosequetly, ptch ca be estmated for a etre segmet. Addtoally, parttog fucto s f ca be used as a scalg factor for the ptch estmato. Scalg may cause reducto of a traset compoet effect o the ptch estmate ad cosequetly better ptch estmato performace. Autocorrelato fucto wth a segmet ca be estmated as: ˆ ρ = s f ρ (9) s s + s + = s where s s a -th segmet oset locato. Aalogcally to the stadard autocorrelato method ptch ca be the estmated accordg to the formula: f s fˆ = m[ ρˆ ] where f s s a sample rate ad m[ ρˆ ] s a autocorrelato estmate pea locato. A llustrato of the ptch estmato wth segmets s show Fgure 3. The crcles dcate ptch estmated correctly. f [ Hz] ˆ Fgure 3. Ptch estmato wth segmets. Subsequetly, ptch ca be estmated precsely for each leap wth a segmet. Accurate ptch estmates are determed upo locatos of local autocorrelato peas earest to the global pea locato of autocorrelato estmate ρˆ. I Fgure 3 correct ptch estmates were dcated. A effect of erroeous estmate ca also be observed case of accurate ptch estmato wth tegrato cosequetly. I the ext sectos a soluto teded to crease ptch estmato accuracy based o ptch predcto s preseted. 4. PREDICTIVE SUPPORT FOR MUSIC TRASCRIPTIO A eural musc predctor [3] was developed as a ptch estmato supportg ut. The appled soluto s based o the Shao s cocept of predctve data codg, employed by Morad ad others for tests wth the Eglsh text [9]. A bloc dagram of eural musc predctve ecoder s preseted Fgure 4. The data s collected wth a buffer. The predctor guesses the ext ote upo the collected data stored a buffer. Thereafter a predcto process s repeated utl a predcted value match actual ote. I case of eural predctor succeedg predctos are emulated by etwors actvato output values. put data e buffer z, e z+, K, talzato e + coder e predctor ˆ + e e e ˆ = + + yes = + o Fgure 4. Musc predctve ecoder. 4. Represetato ad accumulato of data Musc data represetato s essetal for the predcto performace. Three ma ptch represetato techques were examed: bary, modfed Hörel s method ad

4 modfed Mozer s method. All represetato methods employed represet a relato betwee the actual ptch ad a ptch of a precedg ote. I case of bary represetato ptch relato s coded usg a vector of 7 bts as show for a example Tab.. Table. A example of bary ptch relato codg. -octave terval sze (semtoes) octave Orgal Hörel s method was teded for datoc terval codg [5]. The method was the modfed to allow chromatc terval codg. Each ptch relato s the coded usg parameters as show Table. Table. Modfed Hörel s terval represetato. terval - drecto octave semtoes bts bt terval sze represetato Mozer s ptch represetato method characterze ptch as a absolute value. Therefore a modfcato of Mozer s represetato was troduced to allow relatve represetato of terval sze. A represetato was also ehaced by addg drecto parameter ad octave bt. Cotrary to upolar bary ad modfed Hörel s represetato, modfed Mozer s represetato s bpolar. Itervals are coded as show Table 3. Data ca be accumulated usg two types of a buffer: fxedsze buffer ad fadg memory model [5]. I case of fxedsze buffer (where z s a buffer sze) data for z evets (otes) s accumulated usg z dvdual represetato vectors. Fadg memory model allows codg of z evets usg a sgular represetato vector where actual ad faded precedg vector values are accumulated accordg to the formula: b = e r = Table 3. Modfed Mozer s terval represetato. terval - octave drecto semtoes bt terval sze where b deotes buffer values for the -th evet, e deotes the -th evet ad r a fadg factor wth the rage of (, ). 4. Predctor mplemetato The predctor was mplemeted usg Stuttgart eural etwor Smulator (SS) [6]. Three ptch represetato methods preseted above were examed terms of predcto effcecy. For the fxed-sze buffer ts sze was set to 5, ad samples (musc evets). For the fadg memory model fadg factor r was set accordgly to r =.;.5;.8. Expected output of the traed predctor { } should correspod to a subsequet evet e + usg a selected represetato method. Data for the musc predcto expermets were collected from the MIDI database cosstg of fugues from Das Wohltempererte Klaver by J. S. Bach. eural etwors were traed usg all parts wthout the uppermost voce. The part left was used for predctor performace tests. For the expermets a feed-forward eural etwor model wth a sgle hdde layer was used. Predcto effcecy was tested usg the followg measures: correctess at the frst guess, average umber of guesses requred for correct predcto ad lower ad upper F etropy bouds gve as: M ( q q ) F = where M + q log = log q q deotes frequecy of correct aswers at the -th guess ad M s the umber of possble output patters. Accordg to the terval represetato characterstcs subsequet guesses ca be characterzed as a vector cosstg of all possble output represetato vectors ordered terms of matchg measure: om = e+, em, = where m {,, K,M} ˆ (3) =, s a represetato parameter mar, deotes umber of represetato parameters, e ˆ + s a predcted output ad e m s a cosdered possble output patter. umber of guesses requred for predcto ca the be fgured out by locatg expected output patter wth a set of ordered possble output patters. Based o the test results the followg ptch predcto characterstcs ca be deduced. The fest predcto accuracy (correct predcto rate for the frst guess more tha.97) was obtaed for the modfed Hörel s represetato ad fxed-sze buffer. For the fadg memory type buffer predcto effcecy was sgfcatly lower, however fadg memory represetato requres cosderably less computg power for trag ad evaluato. Modfed Mozer s terval codg s ot effcet for musc predcto. 4.3 Predctve support for ptch estmato Evaluated musc predctor was mplemeted to support ptch estmato. Bloc dagram of the system supportg fudametal frequecy estmato wth a sgal segmet x s preseted Fgure 5.

5 x autocorrelato estmate eural musc predctor r data buffer ρˆ P buffer talzato w p eergy weghtg Pˆ Σ maxmum detecto f Fgure 5. Bloc dagram of predcto-supported ptch estmator. Ptch s estmated a followg way. Sgal s aalysed wth segmets. For each segmet a autocorrelato fucto s estmated ad ormalzed. Eergy weghted autocorrelato fucto s gve as: w ˆ ρ ˆ ρ = (4) max ˆ ρ where ρˆ s a autocorrelato fucto maxmum wth -th bad ad s a dcator of a semtoe sub-bad. At the begg predctor data buffer s talsed wth zero values, thus for tal frames the predctor s ot used for supportg ptch estmato. However, ptch values of cosecutve otes estmated o the bass of autocorrelato fucto pea locato are stored wth a buffer. After a certa amout of data was accumulated wth the buffer the predctor starts calculatg probable ptch values for the forthcomg otes. A predctor output vector P (see represetato methods secto 4.) s dstrbuted amog semtoe-wde subbads, scaled usg a weghtg factor w p ad added to curret autocorrelato fucto values. Ptch f ˆ s the estmated o the bass of pea locato of the resultat fucto. Ptch predctor s mplemeted to adust ptch estmate f ˆ wth -th segmet. Ital expermets dcate, that f the autocorrelato fucto peas appear at the edge of two adacet sub-bads ptch estmato may fal. Therefore, ptch predctor output s dstrbuted amog the other sub-bads as follows: p + p+ m p = (5) = + where p s the -th output vector elemet. Accordgly, ptch a -th segmet ca be estmated as follows: + + ˆ ˆ ρ p p Ρ = + w (6) p max ˆ ρ = + where w p s a weghtg factor. As oted before sgal segmetato module may fal case of smooth ter-ote trasets. Hece, whle usg fxed-sze type buffer a ucotrolled data shft wth a sequece may occur. For ths reaso a predctve supported ptch estmator the fadg memory model was used. eural predctor has bee traed usg parts from Fatases for Oboe Solo by G. Ph. Telema excludg the excerpt used for the expermets. 4.4 Ptch estmato performace Ptch estmato performace was aalyzed usg df pr measure. A comparso of autocorrelato-based ptch estmator performace wth ad wthout sgal parttog s show Fgure 6. dfpr [%] stadard parttog,,,3,4,5,6,7,8,9 Fgure 6. Ptch estmato performace wth ad wthout parttog. Subsequetly, ptch estmato tests for the system corporatg eural musc predctor were performed. Expermets were performed for the eural predctors cotag oe hdde layer of 5 ad uts (dcatos x5 ad x accordgly) ad two hdde layers of 5 uts (dcato x5). Tests were performed for varable fadg memory coeffcet r = {.;.5;.8}. Based o the results obtaed wthout usg predcto support, lear atteuato fucto a coeffcet was set to =.8. Frst the correlato betwee predctor output ad the ptch terms of musc cotext was aalyzed. It has proved, that the predctor s output correspods to musc. However, t has bee foud that the predctor ca forecast a dfferet ptch value tha the oe foud the musc. Such a stuato s maly caused by a lmted umber of patters wth learg sets. I other cases predctor ca dcate a few possble solutos. I such a case the choce of a represetato method s very mportat sce some cases (.e. Mozer s method) t ca be umaageable to decode such formato. It s also very mportat to avod fluece of predcto errors o the estmate of followg otes. I such a case a predctor may be lost ad start composg t s ow data stream. To eep the predctor o trac lower fadg memory model factor values ca be used. The predctor has bee the coected wth the ptch estmato system to perform expermets regardg cooperato betwee sgal aalyss ad ptch predcto. Expermets were performed usg varable values of weghtg factor w p ad varable values of fadg memory factor. Itally, the systems behavour was aalyzed usg the segmets for whch the ptch was erroreously estmated wthout usg the predctor. It has bee foud out, that the predctor output ca correspod wth the muscal cotets. Cosequetly, addg weghted predctor s output to autocorrelato fucto estmate ca alter dstorted peas relevat to the actual ptch. Accordgly, expermets wth loger musc sgal were performed usg a excerpt from the Fatasa for oboe solo by G. Ph. Telema [4]. The maxmum ga of ptch predcto accuracy by usg ptch predcto was about

6 percet pots terms of df pr measure (9.% accuracy wthout ad 9.8% accuracy wth ptch predcto support). However, system parameters has to be carefully adusted. I case of mproper adustmets (fadg memory coeffcet, weghtg factor w p etc.) the predctor teds to dmsh ptch estmato accuracy. For example, f the w p factor value s too hgh, the elaborated system teds to geerate musc coarsely related to the aalyzed musc excerpt. 5. COCLUSIOS Upo the performed expermets ad the results preseted followg coclusos were deduced: segmetato of sgal usg the troduced method ca sgfcatly crease ptch estmato accuracy, eural predcto support for ptch estmato ca furthermore crease accuracy. Tag to accout sgal characterstcs (artculato, rapd tempo, reverberato etc.) the obtaed ptch estmato accuracy (maxmum df pr value more tha 9%) should be cosdered as hgh. It should be also oted, that the referece ptch sequece was adusted by matchg dvdual ote durato by audtory comparso wth the recorded excerpt used for the expermets. The referece patter adustmet techque mght cause addtoal ptch estmato errors. The preseted ptch estmato ehacemet techque corporatg segmetato ad predcto ca also be mplemeted usg aother fudametal frequecy estmators such as comb-flter, cepstral or Medds-Hewtt model-based methods. Acowledgmets Research was subsdzed by the Foudato for Polsh Scece ad by the Commttee for Scetfc Research, Warsaw, Polad. Grat o. 4 TD 4. REFERECES [] Beauchamp, J. W., Estmato of Muscal Ptch from Recorded Solo Performaces, Proc. of the 94th AES Coveto, Berl, 6-9 March 993. [] Coo, P. R., Morll, D., Smth, J. O., A Automatc Ptch Estmato ad MIDI Cotrol System for Brass Istrumets, Proc. of Specal Sesso o Automatc Ptch Estmato ASA, ew Orleas, ovember 99. [3] Gelfad, S.A., Hearg: A Itroducto to Psychologcal ad Psychologcal Acoustcs, Marcel Deer,. Yor, 998. [4] Hess, W., Ptch Determato of Speech Sgals: Algorthms ad Devces, Sprger Verlag, Berl, Hedelberg, ew Yor, Toyo, 983. [5] Hörel, D., MELOET I: eural ets for Ivetg Baroque-Style Chorale Varatos, Advaces eural Iformato Processg (IPS ), M. I. Jorda, M. J. Kears, S. A. Solla (eds.), MIT Press, 997 [6] Klapur, A., Wde-bad Ptch Estmato for atural Soud Sources wth Iharmoctes, Proc. of 6th AES Coveto, Preprt 496, Much, May 8-, 999. [7] Koste, B., Żwa, P., Wavelet-Based Automatc Recogto of Muscal Istrumet Classes, ISMIR ( prt). [8] McAulay, R.J., Quater, T.F., Susodal Codg, Speech Codg ad Sythess, W. B. Kle & K. K. Palwal (eds.), pp. -3, Elsever Scece B. V., 995. [9] Morad, H., Grzymała-Busse, J.W., Roberts, J. A., Etropy of Eglsh Text: Expermets wth Humas ad a Mache Learg System Based o Rough Sets, Iformato Sceces, 4 (-), pp. 3-47, 998. [] Mozer, M. C., Coectost Musc Composto Based o Melodc, Stylstc, ad Psychophyscal Costrats, Musc ad Coectosm, P. M. Todd & D. G. Loy (eds.), pp. 95-, The MIT Press, Cambrdge, Massachusetts, Lodo, Eglad, 99. [] Paga,., 4 Caprcc op., Alexader Marov, CD, Erato , 99. [] Raber, L., Cheg, M.J., Roseberg, A.E., Goegal, C.A., A Comparatve Performace Study of Several Ptch Estmato Algorthms, IEEE Tras. ASSP 976, 4, pp [3] Szczerba, M., Recogto ad Predcto of Musc: A Mache Learg Approach, Proc. of 6th AES Coveto, Much, May 8-, 999. [4] Telema, G. Ph., Twelve Fatases For Oboe Solo, Hez Hollger, CD, ppo Columba, Deo, 38C37-789, 984. [5] Todd, P. M. A Coectost Approach to Algorthmc Composto, Musc ad Coectosm, P. M. Todd & D. G. Loy (eds.), pp , The MIT Press, Cambrdge, Massachusetts, Lodo, Eglad, 99 [6] Zell, A. u.a., SS Stuttgart eural etwor Smulator User Maual, Ver. 4..