Word and tree-based similarities for textual entailment

Transcription

1 Wod and tee-based similaities fo textual entailment Fank Schilde R&D Thomson Legal & Regulatoy 610 Oppeman Dive Eagan MN 55123, USA Bidget Thomson McInnes Depatment of Compute Science and Engineeing Univesity of Minnesota 200 Union Steet SE Minneapolis MN 55455, USA Abstact To calculate the textual entailment between two sentences in the RTE data, we use a wod-based similaity combined with a tee-based similaity appoach. Fo each appoach, we expeiment with two diffeent metics. In ode to combine the fou diffeent metics we use an SVM tained on the development set. Ou esults show an oveall accuacy of Intoduction The textual entailment challenge is defined as the task of detemining whethe the hypothesis H is entailed by a longe text T. The definition of textual entailment is based on the human capability in eading T and infeing that H is most likely tue. The textual entailment elation holds, fo example, fo the following pai: T : The settlement must be appoved by Citigoup s boad of diectos and the boad of Regents of the Univesity of Califonia, the lead plaintiff fo investos in the case. H: The settlement is pending appoval by Citigoup s diectos and the boad of Regents at the Univesity of Califonia. The textual entailment elation does not hold, on the othe hand, fo the following pai: T : Asenal sneaked a 1-0 victoy ove Bimingham at Highbuy, taking advantage of Stephen Clemence s own-goal in the 81st minute. H: Asenal lost to Bimingham. Fo this yea s challenge, a development data set of 800 pais was povided by the oganizes. This set is divided into fou subsets. The subsets ae ceated fo diffeent applications and patly taken fom the output fom actual NLP systems. The fou application settings ae infomation extaction (IE), infomation etieval (IR), question answeing (QA), and multi-document summaization (SUM). Ou appoach is patly motivated by the fou diffeent application settings. We hypothesized that the subtasks IE and IR ae bette addessed by a wodbased similaity appoach, wheeas QA and SUM would benefit moe fom a tee-based appoach that takes the tee stuctue of the geneated pase tee into account. Based on this assumption, we developed two wod-based similaity and two tee- based similaity appoaches. Given the similaity scoes fo each of these metics we used an SVM to compute the best combination of the computed metics, the subtasks and the lengths of T and H. 2 Semantic similaity appoaches In this section, we discuss two wod-based semantic similaity metics to identify the similaity between two sentences as a function of the similaity between the wods in the sentence. How similaities of wods can be measued ae descibed in section 2.1 befoe we descibe in moe detail the two metics between

2 sentences we developed based on (Coley and Mihalcea, 2005). 2.1 Wod-based appoaches to semantic similaity Thee exist a numbe of similaity measues which ely on the position of wods in an is-a hieachy and elatedness measues which do not ely on such a hieachy. Fo the wod-based similaity appoach, we chose to look at seven measues. The similaity measues used can be gouped into two categoies: path length based and infomation content (IC) based. The path length based measues used ae: Wu & Palme, Leacock & Chodoow, and Path. The IC measues used ae: Resnik, Jiang & Conath, and Lin. The elatedness measue used is Lesk. The (Wu and Palme, 1994) similaity measue defined in Equation 1 measues the depth of two concepts in WodNet and the depth of thei least common subsume (LCS). The LCS is the most specific concept two concepts shae as an ancesto. sim wup = 2 depth(lcs(c 1, c 2 )) depth(c 1 ) + depth(c 2 ) (1) The (Leacock and Chodoow, 1998) measue defined in Equation 2 is the negative log of the shotest path between two concepts in a taxonomy (in this case WodNet, (Fellbaum, 1998)) divided by twice the total depth of the taxonomy (D). sim lch = log minpath(c 1, c 2 ) 2 D (2) The Path measue (Pedesen et al., 2004) is a simple measue that detemines the similaity between two concepts in WodNet by counting the numbe of nodes between them. The (Resnik, 1995) measue defined in Equation 3 is based on the infomation content. It is the negative log of the pobability of the concepts which is define as the infomation content of the LCS of the two concepts. sim es = IC(lcs(c 1, c 2 ) = log(p (lcs(c 1, c 2 ))) (3) The (J. Jiang, 1997) measue defined in Equation 4 is based on the IC of the two concepts as defined by (Resnik, 1995). The measue is modified to include the length of the path between the two concepts. 1 sim jcn = IC(c 1 ) + IC(c 2 ) 2 IC(lcs(c 1, c 2 )) (4) The (Lin, 1998b) measue defined in Equation 5 is also based on (Resnik, 1995) IC but is modified to include the IC of the two concepts. sim lin = 2 IC(lcs(c 1, c 2 )) IC(c 1 ) + IC(c 2 ) (5) The (Lesk, 1986) measue is based on the ovelap between the wods in the two concepts definitions (in this case WodNet glosses). 2.2 Computing wod similaity fo sentences In this section, we popose two metics to detemine the similaity between two sentences based on the similaity between the wods in the sentences. The fist is a non-symmetic metic that computes the Euclidean distance between a vecto of similaity scoes and the oigin (OVA). The second one is a symmetic metic that computes the Euclidean distance between two vectos containing similaity scoes (TVA) Oigin vecto appoach In the oigin based appoach (OVA), fo each wod in the text sentence, we obtain the maximum similaity scoe between the text wod and the wods with the same coesponding pat-of speech in the hypothesis sentence using the similaity and elatedness measue descibed above. The similaity scoes ae obtained using the WodNet::Similaity package (Pedesen et al., 2004). The measues ae defined fo concept-concept similaity but can be used fo wod-wod similaity as well. Wod- Net is a lexical efeence system which models the hieachical elationships between wods. The similaity package equies that each wod has a pedefined WodNet sense. To obtain these, we use the WodNet::SenseRelate::AllWods (Michelizzi, 2005) package which assigns a WodNet sense to evey content wod in a sentence. This appoach pefomed bette than using just the fist sense of the wod.

3 WodNet is only defined fo noun and vebs. Hence, fo adjectives and advebs, we pefom an exact lexical matching giving them a maximum similaity scoe of one fo a match and zeo othewise. This equation is simila to the maxsim equation poposed by (Coley and Mihalcea, 2005). These computations give us a vecto of maximum similaity scoes which we weight by multiplying the maximum similaity scoe fo the text wod with its Invese Document Fequency (IDF), as defined in Equation 6 whee t k is a wod in the text sentence (cf. (Coley and Mihalcea, 2005)). a a x Inset Λ x Delete x Λ a x a sim tk = maxsim(t k) IDF (t k ) t i maxsim(t i ) IDF (t i ) (6) The Euclidean distance using Equation 7 is calculated between the similaity vecto and the oigin. The close to the oigin the vecto is the less simila the text and hypothesis sentences ae fom one anothe. simscoe(t i, T j ) = sim 2 t k (7) t k Two vecto appoach In the two vecto appoach (TVA), we ceate a similaity vecto fo the text sentence as descibed in ou pevious appoach. We then pefom the same calculations only evesing the text and hypothesis sentences, obtaining a similaity vecto fo the hypothesis sentence. We then calculate the Euclidean distance using Equation 8 between the text vecto (t) and the hypothesis vecto (h). The size of the t and h vectos should be equal, as equied by the metic. Theefoe, we pad the smalle of the two vectos with zeo s. The smalle the distance between the two vectos the moe likely it is that they ae elated. simscoe(t i, T j ) = (sim tk sim hk ) 2 w k (8) 3 Tee-based appoaches to semantic similaity In ode to compute the similaity between the pase tees, we used two tee similaity measues found a x Change x y a y Figue 1: The thee editing options in the liteatue. The fist one is based on the algoithm developed by (Shasha and Zhang, 1989) and has aleady been used in the context of the textual entailment challenge by (Kouylekov and Magnini, 2005). The second one is an appoximate tee similaity metic poposed by (Augsten et al., 2005). Fo the emainde of this pape, these metics will be called fdist and adist, espectively. The tee stuctues used fo these similaity metics wee deived fom dependency pases we obtained fom unning the MiniPa pase on all T H pais (Lin, 1998a). In the following sections, we will biefly descibe the two algoithms. Howeve, due to the limited space we efe the eade to the detailed desciption of these algoithms in the liteatue. 3.1 fdist The oiginal tee edit distance algoithm allows fo thee diffeent opeations in ode to tansfom a tee t into a tee h: (a) delete, (b) inset and (c) change. Figue 1 depicts these thee opeations. The weights of the penalties imposed by the thee diffeent opeations ae defined in table The function nodes(t ) geneates the set of all nodes and leaves in a tee T. The function mi-penalty(x,y) gives back a penalty based on the mutual infomation fo the wods x and y.

4 X Λ : 0 { 0 if x nodes(h) Λ x : 5 othewise x y : mi penalty(x, y) Table 1: Penalty fo editing opeations Note that ou appoach diffes fom the one poposed by (Kouylekov and Magnini, 2005). Instead of elying on the IDF measues, we used mutual infomation (mi) as an infomation theoetic measue fo the similaity between wods to be changed. This measue has shown good esults fo the automatic discovey of synonyms (Tuney, 2001). Instead of mining the web, as (Tuney, 2001) did, we extacted the mutual infomation fo wod pais fom a lage news copus. Incopoating this measue into the change penalty of the tee similaity algoithm will allow us to model the tansfomation fom T to H based on the shaed infomation of the wods. The function mi penalty is defined in a way that wods that nomally occu in simila contexts will get a low penalty (e.g. bought/puchased), wheeas wods that do not shae the same context will get a vey high penalty. While expeimenting with the tee-edit distance algoithm, we noticed that one dawback of this appoach is that the opeations ae caied out without taking the context into account. Passive sentence o tansposed sentences would get an unusual high penalty. Hence, we wee looking fo a tee similaity algoithm that focuses moe on the actual stuctue of the tee. 3.2 adist The appoximate tee edit distance algoithm poposed by (Augsten et al., 2005) has a numbe of advantages ove the oiginal tee edit distance algoithm by (Shasha and Zhang, 1989). Diffeences in the actual tee stuctue become moe ponounced and it is computationally fa less expensive. The algoithm uses so-called p, q-gams which ae computed on the basis of an extended tee T p,q that contains exta empty nodes (see figue 2). The p, q- gams ae deived fom all possible subtees of T p,q given an ancho node that has p 1 ancestos and q childen. In the extended tee in figue 2, the node * bought * * John ca * * * * * * * a * * * * * Figue 2: p, q-extended tee with p = 2, q = 3 bought would be a possible ancho that has as ancesto and combined with the bag of all possible q childen poduces the following p, q-gams: {[, bought,,, John], [, bought,, John, ca], [, bought, John, ca, ], [, bought, ca,, ]}. By geneating all possible p, q-gams fo two tees we obtain a pofile P p,q (t) fo a given tee t. Given a pofile fo each tee, the similaity scoe fo two tees t, h is computed as follows: adist(t, h) = 1 2 Pp,q (t) P p,q (h) P p,q (t) P p,q (h) 4 Combining wod-based and tee-based appoaches (9) Afte obtaining all the scoes fo the wod-based and the tee-based appoaches, we used these featues fo taining an SVM. In addition to the similaity scoes, we added the length of the text and hypothesis as well as the subtask as additional featues. We used WEKA fo taining the SVM using the standad polynomial kenel povided by this machine leaning envionment (Witten and Fank, 2005). 5 Results We submitted two uns vaying the two diffeent wod-based methods. The fist un uses the wodbased oigin vecto appoach (OVA) combined with the two tee-based appoaches adist and fdist. The second un uses the wod-based two vecto appoach (TVA) again combined with the two tee- based appoaches. The esults fo the test set can be found in figue 2. The esults ae within the ange of accuacy esults obtained by systems fo last yea s challenge

5 All IE IR QA SUM RUN 1 Acc RUN 2 Acc Table 2: Results fo test set Run1 and Run 2 (i.e. best paticipating system: 0.586). In paticula, the esults fo the subtasks QA and SUM seem to be encouaging. On the othe hand, the esult fo IE is fa below the baseline (i.e. 0.5). 6 Analysis Ou assumption that the wod-based appoaches combined with the tee-based ones would lead to an impovement of the entie accuacy did tun out to be only patly tue. Moeove, the esults fo the test set tuned out to be quite a bit lowe than what we had obtained fo the taining set. The accuacy fo the taining set can be found in figue 3. All IE IR QA SUM RUN 1 Acc RUN 2 Acc Table 3: Results fo development set A close look eveals that the dop in pefomance is highly subtask-dependent. Wheeas IE and SUM show a vey high dop in accuacy (i.e. up to 0.215), IR and QA showed vitually no o only a modest decease in accuacy (i.e ). The bad pefomance fo the IE subtask may esult fom the fact that the tee-based appoaches match the oot of the pase tees which woks fine fo the following taining example, whee T does not entail H: T : ECB spokeswoman, Regina Schuelle, declined to comment on a epot in Italy s La Repubblica newspape that the ECB council will discuss M. Fazio s ole in t he takeove fight at its Sept. 15 meeting. Appoach Acc. OVA TVA ADIST FDIST A-FDIST ADIST + FDIST + OVA (RUN 1) ADIST + FDIST + TVA (RUN 2) Table 4: Individual esults fo the development set H: Regina Shuelle woks fo Italy s La Repubblica newspape. The test set, on the othe hand, seemed to contain many example whee the H had to be infeed fom a subtee of the pase tee (e.g. a elative clause). T : The Bitish ambassado to Egypt, Deek Plumbly, told Reutes on Monday that authoities had compiled the list of 10 based on lists fom tou companies and fom families whose elatives have not been in contact since the bombings. H: Deek Plumbly esides in Egypt. Since told and eside bea a high penalty, ou system does not deive the entailment elation via the teebased appoaches no do the wod-based appoaches establish a semantic similaity between this vey shot H and the elatively long T. The pediction we made that a combination of diffeent appoaches to semantic similaity would be beneficial was not tue fo the test set, even though the esults fo the development set seemed to indicate that. The esults shown in table 4 descibe an impovement of system s accuacy when diffeent appoaches ae combined. The same numbes obtained afte submitting ou two uns, howeve, show a diffeent pictue fo the test set. The esults of the individual appoaches in table 5 pove that a combination of wod-based and tee-based similaity measues did actually hut accuacy. The combined tee-based appoach (A- FDIST) would have poduced a bette pefomance fo the test set (i.e ). Analyzing the esults fo the taining set, we noticed that the length featue fo T and H alone could

6 give supisingly good esults. A tee stump tained only on these two featues obtained an accuacy of This seems to be an atifact that should be avoided by poducing balanced taining and test sets wt. sentence length in the futue. Refeences Nikolaus Augsten, Michael H. Böhlen, and Johann Gampe Appoximate matching of hieachical data using pq-gams. In Poceedings of Vey Lage Data Bases (VLDB) Confeence, pages ACM Pess. Coutney Coley and Rada Mihalcea Measuing the semantic similaity of texts. In Poceedings of the ACL Wokshop on Empiical Modeling of Semantic Equivalence and Entailment, pages pp , Ann Abo, Michigan. Chistiane Fellbaum WodNet An electonic lexical database. MIT Pess, Cambidge, Massachusetts and London, England. D. Conath J. Jiang Semantic similaity based on copus statistics and lexical taxonomy. In Poceedings on Intenational Confeence on Reseach in Computational Linguistics, Taiwan, pages pp Milen Kouylekov and Benado Magnini Recognizing textual entailment with tee edit distance algoithms. In Oen Glickman Ido Dagan and Benado Magnini, editos, Poceedings of the PASCAL Challenges Wokshop on Recognising Textual Entailment. C. Leacock and M. Chodoow Combining local context and wodnet similaity fo wod sense identification. pages Michael Lesk Automatic sense disambiguation using machine eadable dictionaies: how to tell a pine cone fom an ice ceam cone. In SIGDOC 86: Poceedings of the 5th annual intenational confeence on Systems documentation, pages ACM Pess. Appoach Acc. IE IR QA SUM OVA TVA ADIST FDIST A-FDIST RUN RUN Table 5: Individual esults fo the test set Ted Pedesen, Siddhath Patwadhan, and Jason Michelizzi Wodnet::similaity - measuing the elatedness of concepts. In Daniel Macu Susan Dumais and Salim Roukos, editos, HLT-NAACL 2004: Demonstation Papes, pages 38 41, Boston, Massachusetts, USA, May 2 - May 7. Association fo Computational Linguistics. Philip Resnik Using infomation content to evaluate semantic similaity in a taxonomy. In IJCAI, pages D. Shasha and K. Zhang Fast paallel algoithms fo the unit cost editing distance between tees. In SPAA 89: Poceedings of the fist annual ACM symposium on Paallel algoithms and achitectues, pages , New Yok, NY, USA. ACM Pess. Pete D. Tuney Mining the web fo synonyms: PMI-IR vesus LSA on TOEFL. In Poceedings of the Twelfth Euopean Confeence on Machine Leaning (ECML-2001), pages , Feibug, Gemany. Ian H. Witten and Eibe Fank Data Mining: Pactical machine leaning tools and techniques. Mogan Kaufmann, San Fancisco. Z. Wu and M. Palme Veb semantics and lexical selection. Dekang Lin. 1998a. Dependency-based evaluation of MINIPAR. In Wokshop on the Evaluation of Pasing Systems, Ganada, Spain, May. Dekang Lin. 1998b. An infomation-theoetic definition of similaity. In Poc. 15th Intenational Conf. on Machine Leaning, pages Mogan Kaufmann, San Fancisco, CA. J. Michelizzi Semantic elatedness applied to all wods sense disambiguation. Maste s thesis, Univesity of Minnesota, Duluth, Duluth.