Committee-based Decision Making in Probabilistic Partial Parsing

Transcription

1 Committee-based Decision Making in Probabilistic Partial Parsing * * INUI Takashi and INUI Kentaro * Kyushu Institute of Technology PRESTO,Japan Science and Technology Corporation

2 Background Tree banks Increasing availability of large tree banks Success of statistical approaches to parsing However, Improvements appear to be getting saturated Two new directions for extending the current probabilistic parsing techniques, Probabilistic Partial Parsing Committee-based decision making

3 Overview of today s s talk A probabilistic extension of partial parsing Committee-based probabilistic partial parsing 1. Probabilistic voting 2. Standardization 3. Multiple voting Experiments

4 Bunsetsu phrase (BP) In this talk The target language of experiments is Japanese However, Our proposal is not limited to Japanese It should be able to be applied to other languages like English Japanese.. English....

5 Bunsetsu phrase (BP) A Bunsetsu phrase (BP) is a chunk of words consisting of a content word (noun, verb, etc.) accompanied by some functional words (particle, auxiliary, etc.) A Japanese sentence can be analyzed as a sequence of BPs, which constitute an inter-bp dependency structure Taro goes to school. : Bunsetu phrase /taro /ga /gakkou /ni /iku noun auxiliary noun auxiliary verb

6 Bunsetsu phrase (BP) A Bunsetsu phrase (BP) is a chunk of words consisting of a content word (noun, verb, etc.) accompanied by some functional words (particle, auxiliary, etc.) A Japanese sentence can be analyzed as a sequence of BPs, which constitute an inter-bp dependency structure Taro goes to school. /taro /ga /gakkou /ni /iku noun auxiliary noun auxiliary verb

7

8 Overview Probabilistic extension ( Jensen et al.,1993) Output only a part of the parse tree that are probabilistically highly reliable Input BP2 BP3 Parser Output BP2 BP3 reliable unreliable reliable

9 Overview Probabilistic extension ( Jensen et al.,1993) Output only a part of the parse tree that are probabilistically highly reliable Dependency probabilities (DPs) Input Output 0.90 BP2 BP3 Parser BP BP3 Selecting only dependency relations whose estimated probability is higher than a certain threshold.

10 Overview Probabilistic extension ( Jensen et al.,1993) Output only a part of the parse tree that are probabilistically highly reliable Input Output = 0.7 BP2 BP3 Parser BP BP Selecting only dependency relations whose estimated probability is higher than a certain threshold.

11 C-A A curve coverage = accuracy = # of the decided relations # of all the relations in the test set # of the correctly decided relations # of the decided relations accuracy You can achieve significantly higher accuracy only by sacrificing coverage very little coverage

12 Advantages The user can make a fine-graind arbitrary choice on the trade-off between coverage and accuracy Such trade-off choice makes the existing parsers of wider application accuracy bootstrapping paraphrasing machine translation coverage

13 Estimation of DPs Bottom-up models (Collins, 1996),(Uchimoto et al., 1999) Directly estimate DPs Top-down models (Charniak, 1997), (Shirai et al., 1998) You can estimate DPs, whether BP2BP3 Use the n-best dependency structure candidates coupled you have with a top-down probabilistic model scores. a bottom-up model BP2 BP3 BP2 BP3 BP2 BP BP2 BP st nd rd.

14 Committee-based

15 Overview Committee-based decision making is to combine the outputs from several different systems (e.g. parser) to make a better decision. POS tagging (Halteren et al., 1998; Brill et al., 1999) Parsing (Henderson and Brill, 1999) Word sense disambiguation (Pedersen, 2000) Machine translation (Frederking and Nirenburg, 1994 ) Speech recognition (Fiscus, 1997) These works empirically demonstrated that Committee-based combining different systems often achieved significant improvements over the previous best system.

16 A basic scheme M1 BP2BP3 Committee Committee-based Mk Mm Models (parsers) BP2BP3 BP2BP3 Inputs CF BP2BP3 Output Simple Majority function CF : Combining Function

17 A basic scheme M1 BP2BP3 Committee Committee-based Mk Mm BP2BP3 BP2BP3 CF BP2BP3 Output To realize partial parsing on Simple this scheme, Majority function Models the (parsers) committee Inputs would need to accept probabilistically CF : Combining annotated Function votes

18 Extension (1) : Probabilistic voting M1 BP2BP Committee Committee-based Mk Mm Models (parsers) BP2BP3 BP2BP CF BP2BP3 Output 0.65 Weighted majority function Most Inputs statistical parsers can be members of such a probabilistic CF voting : Combining committee Function

19 Extension (2) : Standardization Committee-based Reliability of dependency probabilities(dps) equally reliable? Ma BP2BP Mb 0.80 BP2 BP3 Reliability of DPs may differ depending on parsers Standardization of DPs

20 Extension (2) : Standardization M BP2BP3 Committee Committee-based Mk Mm Models (parsers) 0.70 BP2BP BP2BP3 Inputs CF 0.65 BP2BP3 Output To standardize input DPs, We add weighting functions

21 Extension (2) : Standardization M BP2 BP3 WFM BP2 BP3 Committee-based Committee Mk Mm Models (parsers) 0.70 BP2 BP BP2 BP3 Inputs WFMk WFMm 0.72 BP2 BP BP2 BP3 CF 0.65 BP2 BP3 Output WF : Weighting Function

22 Extension (3) : Multiple voting Committee-based Each member is allowed to cast (probabilistically parameterized) multiple votes for all the potential candidates BP2 BP BP2 BP probabilistically annotated dependency structure BP2 BP DP matrix DP: Dependency Probability

23 Extension (3) : Multiple voting M BP2 BP3 WFM BP2 BP3 Committee-based Committee Mk Mm Models (parsers) 0.70 BP2 BP BP2 BP3 Inputs WFMk WFMm 0.72 BP2 BP BP2 BP3 CF 0.65 BP2 BP3 Output

24 Extension (3) : Multiple voting Committee-based Committee M1 PM1 WFM1 WM1 Mk PMk WFMk WMk CF O Mm PMm WFMm WMm Output DP matrix Models (parsers) Input DP matrices Weight DP matrices Generalized Committee-based Probabilistic Partial Parsing

25 accuracy Weighting functions A bare DP may not a precise estimation of the actual accuracy Underestimate accuracy C Committee-based D A Overestimate accuracy dependency probability

26 Weighting functions Committee-based You can standardize input DPs by referring to P-A curves acquired from some training data Training data P-A curves

27 Weighting functions : Normal Committee-based Ma 0.80 BP2BP3 BP2BP3 WFMa Normal standardization 0.6 Ma Mb Mb 0 BP2 BP WFMb BP2 BP3 0.60

28 Weighting functions : Class Committee-based Training data 0 Advarbial dep. relations Class-based standardization 0 0 Adnominal dep. relations You P-A could curves prepare may significantly weighting functions differ depending for each on problem class classes

29 Combining function Committee-based CF Weighted majority voting Averaging of the given weight matrices

30 Summary Committee-based Our committee-based scheme: (a)accepts probabilistic parameterized votes as its input (b)accepts multiple voting (c) considers the standardization of original input votes (d)outputs a DP matrix as a final decision DP: Dependency Probability

31 Related works Our voting scheme = Generalization of existing voting techniques for NLP: Probabilistic multiple voting Standardization Committee-based DP matrix output (coverage/accuracy trade-off ) Previous voting techniques Not accept multiple voting POS tagging (Halteren et al., 1998) Parsing (Henderson and Brill, 1999) Not accept probabilistic voting

32 Experiments

33 committee members (parsers) Experiments KANA Ehara, 1998) : a bottom-up model based on maximum entropy estimation CHAGAKE (Fujio et al., 1998) : an extension of the bottom-up model proposed Collins (Collins, 1996) Kanayama s parser (Kanayama et al., 1999) : a bottom-up model coupled with a HPSG Shirai s parser (Shirai et al., 1998) : a top-down model incorporating lexical collocation statistics Peach Pie Parser (Uchimoto et al., 1999) : a bottom up model based on maximum entropy estimation

34 Training / test sets Experiments Kyoto corpus(ver2.0) (Kurohashi et al., 1997) collection of Japanese newspaper articles annotated in terms of : Word boundaries, POS tags BP boundaries, Inter-BP dependency relations Rejected by at least one parser 19,956 sentences 13,990 sentences Assigned inconsistent BP boundaries by different parsers Five-fold cross-validation (for open test)

35 Performance of each individual model Experiments Model (parser) A B C D E Total accuracy point accuracy Total accuracy and 11-point accuracy are both given by C-A curve

36 C-A A curve Experiments accuracy coverage Total accuracy 11-point Total accuracy is the is a accuracy summary of of total the parsing C-A curve, which is is given the accuracy by the average of the case of the accuracy of 11 points where the coverage is 1.0

37 Accuracy of each individual model Experiments Model (parser) A Total accuracy point accuracy Optimal B C Sub-optimal Comparable D E Model A is significantly better than other models

38 Issue (1) : Probabilistic voting Can we easily gather committee members? Shirai s parser(shirai et al., 1998) : a top-down model (not provide DPs directly) By using n-best dependency structure candidates, we were able to estimate DPs reasonably correctly Experiments Yes! accuracy dependency probability Most statistical parsers can be committee members

39 Issue (2) : Standardization Is standardization actually effective? Experiments Yes! 11-point accuracy A : Simple Normal Class 0.96 Simple Normal Class AB AC AD AE ABC ACD 0.92 improved the performance ACE ADE ABCD ABCDE 0.93 Standardization actually BC BD B : BE BCD BDE C : BCDE CD CE CDE A included B included C included committee

40 Issue (3) : Multiple voting Experiments Does multiple voting improve the performance? BP2 BP3 BP2 BP At least Single when voting the size of a committee is small, multiple voting significantly BP2 BP3 outperformed single voting Multiple voting AB AC Small AD AE ABC ACD Yes! ACE multiple voting single voting ADE Committee size ABCD ABCDE Large

41 Issue (4) : Contributions of a committee Does combining parsers actually improve Including the optimal model A, the performance? Not very visible improvement. Including the comparable members such as BC or BD Extensive improvement A : Simple Normal Class Experiments Simple Normal Class C : B : AB AC AD AE ABC ACD ACE ADE ABCD ABCDE BC BD BE BCD BDE BCDE CD CE CDE A included B included C included

42 Issue (4) : Contributions of a committee Does combining parsers actually improve Including the optimal model A, the performance? Not very visible improvement. Including the comparable members such as BC or BD If we have another optimal parser Extensive improvement A : Experiments Simple Normal Class 0.96 Simple Normal Class that was comparable to parser A, then we might achieve significant improvements 0.95 even in case where parser A participates C : B : AB AC AD AE ABC ACD ACE ADE ABCD ABCDE BC BD BE BCD BDE BCDE CD CE CDE A included B included C included

43 Conclusion We proposed a general committee-based framework that can be coupled with probabilistic partial parsing Findings through experiments ( a ) Both multiple voting and vote standardization effectively work in committee-based partial parsing ( b ) If more than two comparably competent optimal models are available, it is likely to be worthwhile to combine them ( c ) Our scheme also enables a non-parametric rulebased parser to make a good contribution