Music Structure based Vector Space Retrieval

Transcription

1 Musc Structure based Vector Space Retreval amunu C. Maddage, Hazhou L Insttute or Inocomm Research (I 2 R) 2, Heng Mu Keng Terrace, Sngapore 963 {maddage, hl}@2r.a-star.edu.sg Mohan S. Kankanhall School o Computng atonal Unversty o Sngapore 543 mohan@comp.nus.edu.sg ABSTRACT Ths paper proposes a novel ramework or musc content ndexng and retreval. The musc structure normaton,.e., tmng, harmony and musc regon content, s represented by the layers o the musc structure pyramd. We begn by extractng ths layered structure normaton. We analyze the rhythm o the musc and then segment the sgnal proportonal to the nter-beat ntervals. Thus, the tmng normaton s ncorporated n the segmentaton process, whch we call Beat Space Segmentaton. To descrbe Harmony Events, we propose a two-layer herarchcal approach to model the musc chords. We also model the progresson o nstrumental and vocal content as Acoustc Events. Ater normaton extracton, we propose a vector space modelng approach whch uses these events as the ndexng terms. In queryby-example musc retreval, a query s represented by a vector o the statstcs o the n-gram events. We then propose two eectve retreval models, a hard-ndexng scheme and a sot-ndexng scheme. Experments show that the vector space modelng s eectve n representng the layered musc normaton, achevng 82.5% top-5 retreval accuracy usng 5-sec musc clps as the queres. The sot-ndexng outperorms hard-ndexng n general. Categores and Subect Descrptors H.3.. [Inormaton Storage and Retreval]: Content Analyss and Indexng - Indexng methods, H.3.3 Inormaton Search and Retreval - Retreval models General Terms Algorthms, Perormance, Expermentaton Keywords Musc structure, beat space segmentaton, harmony event, acoustc event, vector space modelng, n-gram,. ITRODUCTIO Over the past decades, ncreasngly powerul technology has made t easer to compress, dstrbute and store dgtal meda content. There s an ncreasng demand n tools or automatc ndexng and retreval o musc recordngs. The task o musc retreval s to rank a collecton o musc clps accordng to each one s relevance to a query. In ths paper, we are partcularly Permsson to make dgtal or hard copes o all or part o ths work or personal or classroom use s granted wthout ee provded that copes are not made or dstrbuted or prot or commercal advantage and that copes bear ths notce and the ull ctaton on the rst page. To copy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror specc permsson and/or a ee. SIGIR 6, August 6, 26, Seattle, Washngton, USA. Copyrght 26 ACM /6/8...$5.. nterested n musc normaton retreval (MIR) or popular songs that are recorded n the raw audo ormat. In general we am at provdng muscans and scholars tools that search and study derent muscal peces o smlar musc structures (rhythmc structure, melody/harmony structure, musc descrptons, etc); help entertanment servce provders ndex and retreve the songs o smlar tones and semantcs n response to the user queres n the orm o musc clps, whch s also reerred to as query-byexample. The challenges o a MIR system nclude eectve ndexng o musc normaton that supports run-tme quck search, accurate query representaton as the musc descrptor, and robust retreval modelng that ranks the musc documents by relevance score. Many MIR systems have been reported n the survey artcles [8][24]. MIR research communty ntally ocused on developng text based systems where both database and the query are n the MIDI ormat and the normaton s retreved by matchng the melody o query wth the database[5][6][][2][][9][25]. Snce the melody normaton o both query and song database are text based (MIDI), the research has been devoted to database organzaton o the musc normaton (monophonc or/and polyphonc nature) and to textbased retreval models. The retreval models n those systems ncludes dynamc programmng (DP)[5][22][25], n-gram-based matchng [5][6][25] and vector space model[]. Recently, wth the advances n normaton technologes, the communty has started lookng nto developng MIR systems or musc n raw audo ormat. Successul examples towards ths research obectve ncludes the query-by-hummng systems [][22], whch allows a user to nput the query by hummng a melody lne va the mcrophone. To do so, research eorts have been made to extract the ptch contours rom the hummed audo, and to buld a retreval model that measures the relevance between the ptch contour o the query and the melody contours o the ntended musc sgnals. Autocorrelaton [], harmonc analyss [22] and statstcal modelng va audo eature extracton [2] are some o the technques that have been employed or extractng ptch contour rom hummed queres. In [4][9][], xed length audo segmentaton, spectral and ptch contour senstve eatures are dscussed to measure smlarty between musc clps. However, the melody-based retreval model s nsucent or MIR because t s hghly possble that derent songs share an dentcal melody contour. The challenge or MIR o musc n raw audo ormat s to represent the musc content ncludng harmony/melody, vocal and song structure normaton holstcally.

2 Database ndexng Song database Rhythm extracton, Beat Space Segmentaton and slence detecton Musc regon content descrptve eature extracton Harmony descrptve eature extracton Tempo/rhythm cluster (TRC) Acoustc event modelng Harmony event modelng Musc structure normaton extracton and modelng Vector space harmony and acoustc events ndexng st Song TRC-label Frame based ndexng vectors n th song TRC-label Frame based ndexng vectors Indexed musc content n the database Retreval A query clp Query clp TRC-label Frame based ndexng vectors n-gram vector based relevance rankng Retreved song Fgure : Musc structure normaton extracton, vector space content ndexng and retreval In ths paper, we propose novel ndexng and retreval ramework whch descrbes a musc sgnal wth a mult-layer representaton and t s contnuaton o our earler research [5]. We ncorporate tmng normaton o the song wth the musc segmentaton process. Then we detect the progresson o both musc chord and the contents n the musc regons to descrbe the harmony events and the acoustc events respectvely. Inspred by the success o vector space modelng n text-based normaton retreval, we ndex and retreve the songs usng vectors o n-gram statstcs o those events. The proposed ramework s llustrated n Fgure. Ths paper s organzed as ollows. Conceptual musc structure pyramd to vsualze the normaton n the musc structure s dscussed n secton 2. Secton 3 detals the extracton and statstcal modelng o layered musc normaton. In Secton 4, we propose a vector space modelng ramework or MIR wth two retreval models, the hard-ndexng and the sot-ndexng models. In Secton 5, we descrbe the experment results. Fnally, we conclude n Secton MUSIC STRUCTURE As shown n Fgure 2, we represent musc normaton conceptually by a mult-layer pyramd structure. Intro Semantc meanng(s) o the song Verse Chorus Song structure Outro Musc regons -PI, PV, IMV, S {Acoustc events} Harmony /Melody - Duplet, Trplet, Mot, scale, key {Harmony events} Tmng normaton {Bar, Meter, Tempo, notes} Brdge Fgure 2: Layer wse normaton representaton o musc The st layer s the oundaton o the pyramd whch dctates the tmng o a musc sgnal. As tme elapses mxng multple notes together n the polyphonc musc, a harmony lne s created whch s the 2 nd layer o musc normaton. Pure nstrumental (PI), pure vocal (PV), nstrumental mxed vocal (IMV) and slence (S) are the regons that can be seen n a song. PV regons are rare n popular musc. Slence regons (S) are the regons whch have mperceptble musc ncludng unnotceable nose and very short clcks. The content o the musc regons are represented n the 3 rd layer. The 4 th layer and above depcts the semantcs o the song structure, whch descrbes the events or the messages to the audence. Out o all the layers, the most dcult task s to understand the normaton n the top layer, the semantcs o a song rom the song structure pont o vew. In the case o queryby-example MIR, we oten have a partal clp nstead o a ulllength song as a query. Thereore, we beleve that the lower layer musc normaton s more normatve than the top layer as ar as MIR s concerned. As such, the top layer normaton s less crtcal. It s noted that popular songs are smlar n many ways, or example, smlar beat cycle common beat patterns, smlar harmony/melody - common chord patterns, smlar vocal smlar lyrcs and smlar semantc content musc peces or excerpts that creates smlar audtory scenes or sensaton. In ths paper, we wll study the retreval model that evaluates the song smlartes n the aspects o beat pattern, melody pattern and vocal pattern. 3. MUSIC IFORMATIO MODELIG The undamental step or audo content analyss s the sgnal segmentaton where the sgnal wthn a rame can be consdered as quas-statonary. Wth quas-statonary musc rames, we can extract eatures to descrbe the content and model the eatures wth statstcal technques. The qualty o sgnal segmentaton has an mpact on system level perormance o musc normaton extracton, modelng and retreval. Lke n speech processng, earler musc content analyss [][3][8] approaches have used xed length sgnal segmentaton. A musc note can be consdered as the smallest measurng unt o the musc low. Usually smaller notes (/8, /6 or /32 notes) are played n the bars to algn the melody wth the rhythm o the lyrcs and to ll n the gap between lyrcs. Thereore the normaton wthn the duraton o a musc note can be consdered quas-statonary. In ths paper we segment the musc nto rames o the smallest note length nstead o xed length rames. Snce the nter-beat nterval o a song s equal to the nteger multples o the smallest note, ths musc ramng strategy s called Beat Space Segmentaton (BSS). We wll dscuss the musc segmentaton n

3 Secton 3. that captures tmng normaton ( st layer n Fgure 2) o the musc structure. In Secton 3.2 and 3.3 we wll urther dscuss extracton o harmony and musc regon content descrptve eatures that model musc normaton n the 2 nd and the 3 rd layer. 3. Musc Segmentaton and Slence Detecton We llustrate the proposed onset detecton and smallest note length calculaton n Fgure 3. As hghlghted n [5], the spectral characterstcs o the musc sgnals are enveloped proportonal to octaves. So, we rst decompose the musc sgnal nto 8 sub-bands whose requency ranges are shown n Table. Audo musc SB- SB-2 SB-8 Octave scale sub-band (SB) decomposton usng Wavelets Frequency Transents Energy Transents Movng Threshold SO SO 2 SO 8 w 2 SO 8 - Onsets calculated on 8 th sub-band (SB-8) w 8-8 th weght w w 8 ote length estmaton usng crcular autocorrelaton Dynamc Programng Sub-strng estmaton and matchng Fgure 3: Onset detecton and smallest note length calculaton Then the sub-band sgnals are segmented nto 6ms rames wth 5% overlap. Both the requency and energy transents are analyzed usng a method smlar to that n []. We measure the requency transents n terms o progressve dstances n sub-band to 4 because undamental requences (Fs) and harmoncs o musc notes n popular musc are strong n these sub-bands. The energy transents are computed rom sub-band 5 to 8. Table : The requency ranges o the octaves and the sub-bands Sub-band o Octave scale ~B C2~B2 C3~B3 C4~B4 C5~B5 C6~B6 C~B C8~B8 C9~B9 Freq-range(Hz) ~64 64~28 28~ ~52 52~24 24~ ~ ~ ~6384 Eq.() descrbes the computaton o nal onset at tme t, On(t) whch s the weghted sum o sub-band onsets SO r (t). 8 r= On () t = w ( r ). SO () t ( ) r The weght matrx w = {.6,.9,.,.9,.,.5,.8,.6} has been emprcally ound to be the best set or calculatng domnant onsets n musc sgnals. We run crcular autocorrelaton over the detected onsets to estmate the nter-beat proportonal note length. By varyng ths estmated note length, we check or patterns o equally spaced ntervals between domnant onsets On(.) usng a dynamc programmng approach. The most requent smallest nterval, whch s also an nteger racton o other longer ntervals, s taken as the smallest note length. Fgure 4(a) llustrates the process or a -second song clp. The detected onsets are shown n Fgure 4(b). The autocorrelaton o the detected onsets s shown n Fgure 4(c). Inter-beat proportonal smallest note level (83.ms) measure s shown n Fgure 4(d). We assume that the tempo o the song s constant. Thereore the startng pont o the song s used as the reerence pont or BSS. Smlar steps are ollowed or computaton o the smallest note length n the query song clp. However the rst domnant onset s used as the reerence pont to segment the clp back and orth accordngly. The reerence onset s marked n dashed lne n Fgure 4(b). The smallest note length and ts multples orm the tempo/rhythm Smallest note length cluster (TRC). By comparng the TRC o query clp wth TRC o the songs n the database, we can narrow down the search space. (a) Strength (b) Strength (c) Energy (d) Strength second clp rom I am a Lar-Bryan Adams -.5 Detected onsets.5 8 Results o autocorrelaton Inter -beat proportonal smallest note length segments x 5 Sample number (samplng requency = 44 Hz) Fgure 4: seconds clp o the song Slence s dened as a segment o mperceptble musc, ncludng unnotceable nose and very short clcks. We use short-tme energy uncton to detect the slent rames. 3.2 Chord Modelng The progresson o musc chords descrbes the harmony event o musc. A chord s constructed by playng set o notes (>2) smultaneously. Typcally there are 4 chord types (Maor, Mnor, Dmnshed and Augmented) and 2 chords per chord type that can be ound n the western musc. For ecent chord detecton, the tonal characterstcs (Fs, harmoncs and sub-harmoncs) o the musc notes whch comprse a chord should be well characterzed by the eature. Goldsten (93) [] and Terhardt (94) [23] proposed two psycho-acoustcal approaches: harmonc representaton and sub-harmonc representaton, or complex tones respectvely. It s noted that harmoncs and sub-harmoncs o a musc note are closely related to the F o another note. For example, 3 rd and 6 th harmoncs o note C4 are close to F o G5 and G6. Smlarly 5 th and th sub-harmoncs o note E are closed to F o C5 and F#4 respectvely. In our chord detecton system, we place 2 lters centered on Fs o 2 notes n each octave coverng 8 octaves (C2B2 ~C8B8) to capture the strengths o Fs, sub-harmoncs and harmoncs. The lter postons are calculated usng Eq.(2) whch rst maps the lnear requency scale ( lnear ) nto octave scale ( octave ) where Fs,, F req are samplng requency, number o FFT ponts and reerence mappng pont respectvely. We set requency resoluton (Fs/) equal to Hz, F req =64Hz (F o the note C2) and C=2 (2 ptches). The lter (rectangular lter n dashed lne) poston near note G n both octave and lnear requency axs s depcted n Fgure 6. Fs * *log lnear Octave = C 2 mod C * F re ( 2 ) The reasons or usng lters to extract tonal characterstcs o notes are explaned below.. Due to physcal conguraton o the nstruments, the Fs o the notes may vary rom the standard values (A4=44Hz s used as the concert ptch). 2. Though the physcal octave rato s 2:, cogntve experments have hghlghted that ths rato s close at lower requences, but ncreases wth the hgher requences. It exceeds by 3% at about 2 khz [26]. Thereore, we poston the lters to detect the strengths o the harmoncs o the shted notes.

4 2 In our experments, t s ound that the tonal characterstcs n an ndvdual octave can even eectvely represent the musc chord. To model these tonal characterstcs n the octaves, we propose a 2-layer herarchcal model or musc chord (see Fgure 5). The models n the st layer are traned usng ptch class prole (PCP) eature vectors (2-dmensonal) whch are extracted rom ndvdual octaves. Due to poor chord detecton accuracy n the C9B9 octave, only C2B2~C8B8 octaves are consdered. The constructon o PCP vector or n th sgnal rame and or each octave s explaned n Eq.(3). F strengths o the α th note and related harmonc and sub-harmonc strengths o other notes are summed up to orm the α th coecent o the PCP vector. In Eq.(3), S(.) s the requency doman magntude (n db) sgnal spectrum. W (OC, α) s the lter whose poston and the pass-band requency range vares wth both octave ndex (OC) and α th note n the octave (OC). I the octave ndex s, then the respectve octave s C2B2. 2 PCP n OC ( α ) = S(.) W( OC, α ) OC =..., α =...2. ( 3 ) The 2 nd layer model s traned wth the outputs o the st layer models whch are organzed nto a eature vector. In our mplementaton we use 4 Gaussan mxtures or each model n layer and 2. Thereore nput vectors to the layer 2 model are probablstc vectors. Ths 2-layer modelng can be vsualzed as rst transormng eature space represented tonal characterstcs o the musc chord nto probablstc space at the layer and then modelng them at layer 2. We use ths 2 layer representaton to model 48 musc chords n our chord detecton system. n th sgnal rame PCP n OC= PCP n OC=2 PCP n OC= Model OC= Model OC=2 Model OC= Layer Probablstc eature vector Model n Layer 2 eature vectors n octaves Fgure 5: Two layers herarchcal representaton o a musc chord 3.3 Musc Regon Content Modelng As dscussed n Secton 2, PV, PI, IMV and S are the regons types n a song (3 rd layer). However PV regons are comparatvely rare n popular musc. Thereore both PV and IMV regons are consdered as vocal (V) regon. In ths way, we can ust ocus on contents o 3 regons (PI, V and S). Slence detecton has been dscussed n Secton 3.. Sung vocal lne carres more descrptve normaton about the song than other regons. In the PI regons, the extracted eature must be able to capture the normaton generated by lead nstruments (typcally the tunes/melody). To ths end, we examne Octave scale cepstral coecent (OSCC) eature and Mel-requency cepstral coecent (MFCC) eature or ther capabltes to characterze musc regon content normaton. MFCC have been hghly eectve characterzng subectve ptch and the requency spectrum o speech sgnals [4]. OSCCs are computed by usng a lter bank n requency doman. Flter postons n the lnear requency scale ( lnear ) are computed by transormng lnearly postoned lters n the octave scale ( octave ) to lnear usng Eq.(2). We set C=2, F re =64 Hz n the Eq.(2) so that 2 overlappng rectangular lters are postoned n each octave rom C2B2 to C9B9 octave (64 ~ 6384) Hz. The Hammng shape o lter/wndow has sharp attenuaton and t suppresses valuable normaton n the hgher requences nearly by 3 old as compared to the rectangular shape lter [4]. Thereore, a rectangular lter s better than Hammng lter or musc sgnal analyss because they are wde band sgnals compared to speech sgnals. Fgure 6 depcts octave to lnear lter poston transormaton. The output Y(b) o the b th lter s computed accordng to Eq.(4) where S(.) s the requency spectrum n decbel (db), H b (.) s the b th lter, and m b and n b are boundares o b th lter. Yb ( ) = nb SaH ( ) b( a) ( 4 ) a= mb Eq.(5) descrbes the computaton o β th cepstral coecent where k b, and Fn are center requency o the b th lter, number o requency samplng ponts and number o lters respectvely (Fn=2 n our case). Flter postons n octave scale Fn 2π C( β ) Y( b)cos( kb β ) β b = ocave C B A# A G# G F# F E D# D C# C = ( 5 ) ~ B C2 ~ B2 C3 ~ B3 C4 ~ B4 C5 ~ B5 5 F Re 5 Flter poston n Octave scale s mapped to lnear requency scale Frequency band o octaves n lnear requency scale Flter postons n lnear requency scale Hgher octaves Equaton () Fgure 6: Transormaton o octave scale lter postons to lnear requency scale Sngular values (SVs) ndcate the varance o the correspondng structure. Comparatvely hgh sngular values descrbe the number o dmenson n whch the structure can be represented orthogonally. Smaller sngular values ndcate the correlated normaton n the structure and consdered to be nose. We perorm sngular value decomposton (SVD) over eature matrces extracted rom PI and V regons. Fgure shows the normalzed sngular value varaton o 2 OSCCs and 2 MFCCs extracted rom both PI and V regons o a Sr Lankan Song Ma Bala Kale ( ). We use 96 lters or calculatng MFCCs and OSCCs. It can be seen that sngular values o OSCCs are hgher than o MFCCs or both PV and PI rame. The average o 2 sngular values per OSCCs or PV and PI rames are.294 and.325. However, or MFCC, they are as lower as.8 and.93 respectvely. As shown n Fgure, the sngular values are n descendng order wth respect to the ascendng coecent numbers. The average o the last sngular values o OSCCs s nearly % hgher than those o MFCCs, whch means the last OSCCs are less correlated than the last coecents o MFCCs. Thus we can conclude that the OSCCs are less correlated than MFCCs n representng content o musc regons. 6 lnear (Hz)

5 ormalzed sngular values derved rom 2 MFCCs Vocal (V) ormalzed sngular values derved rom 2 OSCCs Coecent number OSCC MFCC 5 Pure Instrumental (PI) Coecent number Fgure : Sngular values rom OSCCs and MFCCs or PV and PI rames. The rame sze s a quarter note length (662ms) 4. MUSIC IDEXIG AD RETRIEVAL Unlke text document that uses words or phrases as ndexng terms, a musc sgnal s a contnuous dgtal sgnal wthout obvous anchors or ndexng. The challenges o ndexng musc sgnal are two old. Frst, what would be good ndexng anchors; second, what would be good representaton o musc contents or ndexng and retreval. In Secton 3, we have dscussed the statstcal modelng o musc normaton n a mult-layer paradgm as llustrated n Fgure 2. Layer-wse normaton representaton allows us to descrbe a musc sgnal quanttatvely n a descrptve data structure. ext, we wll propose two ndexng terms.e. harmony event and acoustc event to descrbe the normaton n the 2 nd and 3 rd layers o the musc structure pyramd. 4. Harmony Event and Acoustc Event Progresson o musc chords descrbes the Harmony Event. In Secton 3.2, we explaned sub-band PCP eature extracton and a 2-layer herarchcal chord modelng. ote that t s relatvely easy to detect beat spacng n a musc sgnal. A beat space s a natural choce as a musc rame, and thus the ndexng resoluton o a musc sgnal. Suppose that we have traned 48 rame-based chord models, as shown n Fgure 5 (4 chord types Maor, Mnor, Dmnsh and Augmented n combnaton wth 2 chords each type). Each chord model descrbes a rame-based harmony event whch can serve as the ndexng term. One can thnk o musc as a chord sequence, wth each chord spannng over multple rames. A chord model space Λ= {C } can be traned on a collecton o chord-labeled data. We use the HTK 3.3 toolbox or tranng such a 2-layer chord model space. At run-tme, a musc rame O n s recognzed and converted to a harmony event Î h, and a musc sgnal s thereore tokenzed nto a chord sequence. Iˆ h = arg max p( on c) =,...,48 ( 6 ) c Pure nstrumental (PI) and the vocal (V) regons contan the descrptve normaton about the musc content o a song. A song can be thought o as a sequence o nterwoven PI and V events, that we call Acoustc Events. We extract 2 OSCC eatures rom each musc rame. We tran two Gaussan Mxture models (GMMs) o 64 mxtures. Each GMM s traned on a collecton o eatures rom one o the two events. We dene the rame-based acoustc events as another type o ndexng term n parallel wth harmony events. Suppose we denote r or PI and r 2 or V event. They are traned rom a labeled database. At run-tme, a musc rame O n s recognzed and converted to a V or PI event Î r. Eq.(6) and () can be seen as the chord and acoustc event decoders. ^ I r = arg max po ( r) ( ) =,2 n We ndex the contents n slence regons (S) wth zero observaton. Inspred by the dea n text categorzaton where we use lexcal words as ndexng terms to orm a document vector or a text document, we attempt to use the events as ndexng terms to desgn a vector or a musc segment. ext let us ormulate the ndexng and retreval problem cast n the vector space model ramework. We wll study two retreval models, hard-ndexng and sot-ndexng n-gram MIR models. 4.2 n-gram Vector The harmony and acoustc decoders serve as the tokenzers or musc sgnal. The tokenzaton process results n two synchronzed streams o events, a chord and an acoustc sequence, or each musc sgnal. An event s represented by a tokenzaton symbol. They are represented n a text-lke ormat. It s noted that n-gram statstcs has been used n many natural language processng tasks to capture short-term substrng constrants such as letter n-gram n language dentcaton [2] and spoken language dentcaton [4]. I we thnk o the chord and acoustc tokens, as the letters o musc, then a musc sgnal s a document o chord/acoustc transcrpts. Smlar to the letter n-gram n text, we can use the token n-gram o musc as the ndexng term, whch ams at capturng the short-term syntax o muscal sgnal. The statstcs o token themselves represent the token ungram. Vector space modelng (VSM) has become a standard tool n textbased IR systems snce ts ntroducton several decades ago [2]. It uses a vector to represent a text document. One o the advantages o the method s that t makes partal matchng possble. We can derve the dstance between documents easly as long as the vector attrbutes are well dened characterstcs o the documents. Each coordnate n the vector relects the presence o the correspondng attrbute, whch s typcally a term. A chord/acoustc token n a musc sgnal s ust lke a term n a document. Inspred by the dea o VSM n text-based IR, we propose usng a vector to represent a musc segment. I a musc segment s thought o as an artcle o chord/acoustc tokens, then the statstcs o the presence o the tokens or token n-grams descrbe the content o the musc. Suppose that we have a token sequence, t t 2 t 3 t 4. We derve the ungram statstcs rom the token sequence tsel. We derve the bgram statstcs rom t (t 2 ) t 2 (t 3 ) t 3 (t 4 ) t 4 (#) where the acoustc vocabulary s expanded over the token s rght context. Smlarly, we derve the trgram statstcs rom the t (#,t 2 ) t 2 (t,t 3 ) t 3 (t 2,t 4 ) t 4 (t 3,#) to account or let and rght contexts. The # sgn s a place holder or ree context. In the nterest o manageablty, we only use up to bgrams. In ths way, or an acoustc vocabulary o c =48 token entres n the chord stream, we have 48 ungram 48 requency tems n n the chord vector n = { n,..., n,..., n } as n Fgure 8. n s equal to, t n =c otherwse t s. Fgure 8: A count vector representaton o a musc rame Smlarly we have 2 ungram requency tems n the acoustc vector or the acoustc stream. For smplcty, we only ormulate n next. To capture the short-term dynamcs, we can easly derve the bgram representaton or two consecutve rames. As such, we buld a chord bgram vector o [48x48=234] dmensons,

6 =,, 48,48 n { n,..., n,..., n }, n where both n = and. n+=, then =; otherwse, n =. Smlarly an acoustc bgram vector o [2x2=4] dmensons can be ormed. For a musc segment o rames, we construct a chord ungram vector 48 = {,...,,..., } by aggregatng the rame vectors wth the th element as = n= n ( 8 ) We can construct ts chord bgram vector o [48x48=234],, 48,48 dmensons n = {,...,,..., } n a smlar way wth the (,) th element as,, n= n = ( 9 ) The acoustc vector can be ormulated n a smlar way wth a 2- dmensonal vector or ungram and [2x2=4] dmensonal vector or bgram. Fgure 9 shows schematcally how an n-gram vector s constructed usng rames o ungram vector and how the relevance score s evaluated between a query and a musc segment. A query (song clp) A song n the database v v + v + Relevance rankng n-gram vector V V Fgure 9: The musc database s ndexed by n-gram vector. Each harmony/acoustc event s assocated wth an ndexng vector. The smlarty between a query and a musc segment n the database s measured or relevance rankng. Although we use 2-dmensonal coordnate or the bgram count, the vector can be treated as a -dmensonal array. The process o dervng ungram and bgram vectors or a musc segment nvolves mnmum computaton. In practce, we can compute those vectors at run-tme drectly rom the chord/acoustc transcrpts resultng rom the tokenzaton. ote that the tokenzaton process compares a musc rame aganst all the chord/acoustc events at a hgher computatonal cost. It can be done o-lne. Followng the text-based IR process, the MIR process computes the smlarty between a query musc segment and all the canddate musc segments. For smplcty, let ( q) denote the u chord ungram vector (48 dmensons) and, ( q) denote the chord bgram vector (234 dmensons) or a query o rames. r Smlarly, a chord ungram vector ( d) and a chord bgram u, vector ( d) can be obtaned rom any segment o rames n the musc database. The smlarty between two n-gram vectors can be dened as r r r r ( q) ( d) s( ( q), ( d)) = r r ( ) ( q) ( d) r r r r,,,, ( q) ( d) s( ( q), ( d)) = r r,, ( ) ( q) ( d) Wth Eq.() and Eq.(), we can rank the musc segments by ther relevance. The relevance s can be dened by the uson o ungram and bgram smlarty scores. 4.3 Expected Frequency o n-gram Although t would be convenent to derve the term count rom token sequences derved rom a musc query, we nd that the tokenzaton s aected by many actors. For example, the tokenzaton does not always produce dentcal token sequence or two smlar musc segments. The derence could be due to the varaton n beat detecton, varaton o musc productons between the query and the ntended musc. The nconsstency between the tokenzaton o the query and the ntended musc produce an undesred msmatch as ar as MIR s concerned. Assumng that the numbers o beats n query and musc are detected correctly, the nconsstency s characterzed by substtutons o tokens between the desred label and the tokenzaton results. I a token s substtuted, then t presents a msmatch between the query and the ntended musc segment. To address ths problem, we propose usng the tokenzers as probablstc machnes that generate a posteror probablty or each o the chord and acoustc events. I we thnk o the n-gram countng as nteger countng, then the posteror probablty can be seen as sot-hts o the events. For brevty, we only ormulate the sot-hts or chord vector. Accordng to Bayes rule, we have ( on c) p( c) pc ( on) = ( on c) p( c) ( 2 ) where p(c ) be the pror probablty o the event c. Assumng no pror knowledge about the events, p(c ) can be dropped rom Eq.(2), whch s then smpled as ( on c) pc ( on) = ( on c) ( 3 ) Let P(c o n ) be denoted as p n. It can be nterpreted as the expected requency o event c at n th rame, wth the ollowng propertes, (a) p n, (b) 48 = pn =. A rame s represented by a vector o contnuous values as llustrated n Fgure 8, whch can be thought o a sot-ndexng approach as opposed to the hard-ndexng approach or musc rame usng n-gram countng. The sotndexng relects how a rame s represented by the whole model space whle the hard-ndexng estmates the n-gram count based on the top-best tokenzaton results. We have good reason to expect sot-ndexng to provde hgher resoluton vector representaton or a musc rame Fgure : An expected requency vector or a musc rame Assumng the musc rames are ndependent o each other, the ont posteror probablty o two events and between two rames, n th and (n+) th can be estmated as, pn = pn p n + ( 4 ),, where pn has propertes smlar to that o p n, (a) p n, 48 48, (b) p = = n =. For a query o rames, the expected requency o ungram and bgram can be estmated as p n= n,, p n= n E{ } = ( 5 ) E{ } = ( 6 ) Thus the sot-ndexng vector or query and matchng musc r r segment are E{ ( q)} and E{ ( d)} respectvely. Replacng

7 ( q) wth E{ ( q)}, ( d) wth E{ ( d)} n Eq.(2) and Eq.(3), the smlar relevance scores can be used or sot-ndexng rankng. 5. EXPERIMETS We rst study the chord and acoustc modelng perormance. Then we carry out MIR experments. We establshed a 3 song database DB (44. khz samplng rate, 6 bts per sample, mono channel) extracted rom musc CDs or MIR experments. Songs n DB are sung by 2 artsts as lsted n Table 2, each on average contrbutng 5 songs. The tempos o the songs are n the rage o 6~8 beats per mnute. Agnetha Fatskog 2. Celne Don 3. Cranberres 4. Ddo 5. Fath Hll Table 2: The artsts n the song database Female Artsts 6. Kathryn Wllams. Madonna 8. Mandy Moore 9. Marah Carey. Shana Twan Male Artsts. Ben Jelen 6. Mchael Bolton 2. Bryan adams. Mchael Jackson 3. Cl Rchard 8. MLTR 4. Elton John 9. Rchard Marx 5. Justn Tmberlake 2. West le 5. Harmony Event Modelng Harmony events are descrbed by the progresson o musc chords. Each o the 48 chord models s a 2-layer representaton o Gaussan mxtures (see Fgure 5) and s traned wth annotated samples n a chord database (CDB). The CDB ncludes recorded chord samples rom orgnal nstruments (strng type, bow type, blowng type, etc) as well as synthetc nstruments (sotware generated). In addton, the CDB also ncludes chord samples extracted rom 4 Englsh songs (a subset o DB), wth the ad o musc sheets and lstenng tests. Thereore we have around mnutes o each chord sample spannng rom C2 to B8. % o the samples o each chord are used or tranng and the rest 3% or testng n cross valdaton setup. Expermental results are shown n Fgure. Average o correct chord detecton accuracy n % 8 6 TLM (Two layer model).28 Beat space segmentaton (BSS) SLM (Sngle layer model) Fxed length segmentaton(fix) Fgure : Average correct chord detecton accuracy The results o the proposed 2-layer model (TLM) are compared wth sngle layer model (SLM). Sngle layer chord model s constructed usng 28 Gaussan mxtures. General PCP eatures vectors (GPCP) are used or tranng and testng the SLMs. th Eq.() explans the computaton o α coecent o the GPCP eature vectors. GPCP n ( α ) n = PCP ( α) ( ) OC = It s noted that the proposed TLM wth eature extracted rom BSS outperorms the SLM approach by 5% n absolute accuracy. 5.2 Acoustc Event Modelng We compare perormance o OSCCs and MFCCs or modelng regons PI and V. SVD analyss depcted n Fgure hghlghts that OSCCs characterze musc content more uncorrelatedly than MFCCs. In ths experment, we selected Englsh songs ( songs per artst and 5 artsts per gender) rom DB. We annotate OC the V and PI regons. Each V and PI class normaton s then modeled wth 64 GMs. songs are used by cross valdaton where 6/4 songs are used as tranng/testng n each turn. Table 3 shows correct regon detecton accuraces or optmzed number o both the lters and coecents o MFCC and OSCC eatures. We report the correct detecton accuracy or PI-regon and V- regon, when the rame sze s equal to both beat space and xed length (3ms). Both OSCC and MFCC perorm better when the rame sze s beat space. As OSCC outperorms MFCC n general, we use t or modelng acoustc events. Table 3: Correct Average classcaton o PI and V regons Featureo. o lters o. o coecents PI(%) -BSS V(%)-BSS Avg(PI+V) %-FIX OSCC MFCC Musc Inormaton Retreval In DB, we select 4 clps o 3-second musc as queres rom each artst n the database, totalng 8 clps. Out o 4 clps, two clps belong to V regon and other two manly belongs to PI regon. For a gven query, the relevance score between a song and the query s dened as the sum o the smlarty score between the top K most smlar ndexng vectors and the query vector. Typcally, we set K to be 3. Ater computng the smallest note length n the query, we check the tempo/rhythm clusters o the songs n the data base. For song relevance rankng, we only consder the songs whose smallest note lengths are n the same range (wth ±3ms tolerance) as the smallest note length o the query or nteger multples o them. Then the survvng songs n the DB are ranked accordng to ther respectve relevance scores. Fgure 2 shows the average accuracy o the correct song retreval when the query length s vared rom 2-sec to 3-sec. Both chord events and acoustc events are consdered or constructng n-gram vectors. Average o the correct retreval n % Length o the query clp n seconds Fgure 2: Average retreval accuracy o songs Retreved song wthn top 5 (T5) Retreved song wthn top (T) The average accuracy o correct song retreval n top choce s around 6% or the query length vares ro 5 ~3-sec. For the smlar query lengths, the retreval accuracy or top-5 canddates s mproved by 2%. In Table 4 we study the chord event eect and the combned eect o chord and acoustc events on the retreval accuracy. Table 4: Eects o chord and acoustc event normaton n MIR Avg accuracy n % T5 T (5sec Query length) Harmony event - I h I h + acoustc event Harmony event - I h I h + acoustc event Sot ndexng Hard ndexng It can be ound that sot-ndexng outperorms hard-ndexng (see Eq.(8), Eq.(9)). In general, combnng acoustc events and chord events yelds a better perormance. Ths can be understood by the act that smlar chord patterns are lkely to occur n derent songs. The acoustc content helps derentate one rom the other

8 6. DISCUSSIO AD COCLUSIO We have proposed a novel ramework or MIR. We vsualze musc normaton (tmng, harmony and musc regon contents) n the orm o a musc structure pyramd. We ncorporate tmng normaton n the beat space segmentaton o musc sgnal. A two-layer herarchcal chord model has been proposed to descrbe the harmony events. Content progresson o nstrumental and vocal regons has also been modeled to descrbe acoustc events. Ater modelng layered musc normaton n the vector space, we explored two retreval models, hard-ndexng and sot-ndexng. Our experments show that octave scale musc normaton modelng ollowed by the nter-beat nterval proporton segmentaton s more ecent than wth the xed length musc segmentaton. We ound the sot-ndexng retreval model s more eectve than the hard-ndexng one. The uson o chord model and acoustc model statstcs mproves retreval accuracy eectvely. The overall expermental results convnce our ocus on layer-wse musc processng and vector space retreval model are promsng research drectons. We nd that musc normaton n derent layers complements each other to acheve an mproved MIR perormance. The robustness n ths retreval modelng ramework depends on how well the normaton s extracted. We wll contnue to ocus on the extracton o uncorrelated musc normaton. Even though musc retreval s the targeted applcaton n ths paper, the proposed vector space musc modelng ramework s useul or developng many other applcatons such as musc summarzaton, streamng, musc structure analyss, and creatng multmeda documentary usng musc semantcs. In the uture, we wll work on extendng t to other relevant applcatons.. REFERECES [] Berenzweg, A., Logan, B., Ells, D.P.W., and Whtman, B. A Large-Scale Evaluaton o Acoustc and Subectve Musc- Smlarty Measures. In Computer Musc Journal, Summer, 24, [2] Cavnar, W.B., and Trenkle, J.M. -Gram-Based Text Categorzaton. In Proc. o 3 rd Annual Symposum on Document Analyss and Inormaton Retreval, 994. [3] Cha, W., Vercoe, B. Structure Analyss o Musc Sgnals or Indexng and Thumbnalng. In Proc. o the ACM/IEEE JCDL, May 23. [4] Deller, J. R., Hansen, J.H.L., and Proaks, H. J. G. Dscrete- Tme Processng o Speech Sgnals, IEEE Press, 2. [5] Dorasamy, S., and Rüger, S. Robust Polyphonc Musc Retreval wth -Grams. In Journal o Intellgent Inormaton Systems. Vol 2, o.. pp 53-, 23. [6] Downe, J.S., and elson, M. Evaluatng a Smple Approach to Musc Inormaton Retreval Method. In Proc. ACM SIGIR, July 2. [] Duxburg. C, Sandler. M., and Daves. M. A Hybrd Approach to Muscal ote Onset Detecton. In Proc. Int. Con. DAFx. Hamburg, Germany, Sept, 22. [8] Foote, J. Vsualzng Musc and Audo Usng Sel-Smlarty. In Proc. ACM MM, Oct, 999. [9] Fushma, T. Real Tme Chord Recognton o Muscal Sound: A System Usng Lsp Musc. In Proc. ICMC, Oct [] Ghas, A., Logan, J., Chamberln, D., and Smth, B. C. Query by Hummng: Muscal Inormaton Retreval n an Audo Database. In Proc. o ACM MM, ov, 995. [] Goldsten, J. L. An Optmum Processor Theory or the Central Formaton o the Ptch o Complex Tones. In JASA, Vol. 54, 93. [2] Kageyama, T., Mochzuk, K., and Takashma, Y. Melody Retreval wth Hummng. In Proc. ICMC, Sept, 993. [3] Lemström, K., and Lane, P. Musc Inormaton Retreval usng Muscal Parameters. In Proc. o the ICMC, Oct, 998. [4] Ma, B., and L, H., A Phonotactc-Semantc Paradgm or Automatc Spoken Document Classcaton. In Proc. o ACM SIGIR, Aug, 25. [5] Maddage C.., Xu, C., Kankanhall, M.S., and Shao, X, Content-based Musc Structure Analyss wth the Applcatons to Musc Semantc Understandng, In ACM Multmeda Conerence, Oct. 24. [6] Mcab, R.J., Smth, L.A., Wtten, I.H., Henderson, C.L., and Cunnngham, S.J. Towards the Dgtal Musc Lbrary: Tune Retreval rom Acoustc Input. In Proc. ACM Dgtal Lbrares, March, 996. [] Melucc, M., and Oro,. Musc Inormaton Retreval usng Melodc Surace. In Proc. ACM Dgtal Lbrares, Aug, 999 [8] Pckens, J. A Survey o Feature Selecton Technques or Musc Inormaton Retreval. Techncal report, Center o Intellgent Inormaton Retreval, Dept. o Computer Scence, Unversty o Massachusetts, 2. [9] Pckens, J. and Ilopoulos, C. Markov Random Felds and Maxmum Entropy Modelng or Musc Inormaton Retreval. In Proc. o ISMIR, Sept, 25. [2] Salton, G. The SMART retreval system. Prentce-Hall, Englewood Cls, J, 9. [2] Shh, H.-H., arayanan, S. S., and Kuo, C.-C. J. An HMM- Based Approach to Hummng Transcrpton. In Proc. o ICME, Aug, 22. [22] Song, J., Bae, S. Y., and Yoon, K. Md-Level Musc Melody Representaton o Polyphonc Audo or Query-by-Hummng System. In Proc. o ISMIR, Oct, 22. [23] Terhardt, E. Ptch, Consonance and Harmony. In JASA, Vol. 55, o. 5, 94. [24] Typke, R., Werng, F., and Veltkamp, R. A Survey o Musc Inormaton Retreval Systems. In Proc. o the ISMIR, Sept. 25. [25] Utdenbogerd, A. L., and Zobel, J. An archtecture or eectve musc normaton retreval. In Journal o the Amercan Socety or Inormaton Scence and Technology, Vol. 55, o. 2, pp. 53-5, 24. [26] Ward, W. Subectve Musc Ptch. In JASA, Vol. 26, 95