Today we will... Foundations of Natural Language Processing Lecture 13 Heads, Dependency parsing. Evaluating parse accuracy. Evaluating parse accuracy

Transcription

1 Today we will... Foundations of Natural Language Processing Lecture 13 Heads, Dependency parsing Alex Lascarides (slides from Alex Lascarides, Henry Thompson, Nathan chneider and haron Goldwater) 6 March 2018 Provide metrics for evaluating a parser Return to the problem of PCFGs uggest a fix This fix leads to an approach out constituent structure! Dependency parsing Alex Lascarides FNLP Lecture 13 6 March 2018 Alex Lascarides FNLP Lecture 13 1 Evaluating parse accuracy Compare gold standard tree (left) to parser output (right): Pro he Vt PosPro N Pro he Vp Pro Vi Evaluating parse accuracy Compare gold standard tree (left) to parser output (right): Pro he Vt PosPro N Pro he Vp Pro Vi her duck her duck her duck her duck Output constituent is counted correct if there is a gold constituent that spans the same sentence positions. Harsher measure: also require the constituent labels to match. Pre-terminals (lexical categories) don t count as constituents. Precision: (# correct constituents)/(# in parser output) = 3/5 Recall: (# correct constituents)/(# in gold standard) = 3/4 F-score: balances precision/recall: 2pr/(p+r) Alex Lascarides FNLP Lecture 13 2 Alex Lascarides FNLP Lecture 13 3

2 F-scores for parsing on WJ corpus: vanilla PCFG: < 80% 1 Parsing accuracies lexicalizing + cat-splitting: 89.5% (Charniak, 2000) Best current parsers get about 92% Numbers get better if we look at top 5 or top 10 However, results on other corpora and other languages are considerably lower. Definitely not a solved problem! ummary Probabilistic models of syntax can help disambiguation and speed in broadcoverage parsing. by computing the probabilities of each tree or sub-tree as the product of the rules in it, and choosing the best option(s). Treebanks provide training data for estimating rule probabilities. However, to do well, we need to be clever: tandard categories in the treebank don t capture some important facts about language. By creating more detailed categories, we can encode more information in the PCFG framework. 1 Charniak (1996) reports 81% but using gold PO tags as input. Alex Lascarides FNLP Lecture 13 4 Alex Lascarides FNLP Lecture 13 5 Recall Problem Vanilla PCFGs No lexical dependencies Replacing one word another the same PO will never result in a different parsing decision, even though it should! vs. binoculars he stood by the door covered in tears vs. he stood by the door covered in ivy stray cats and dogs vs. iamese cats and dogs A way to fix PCFGs: lexicalization Create new categories, this time by adding the lexical head of the phrase (note: N level under s not shown for brevity): V- - PP-binoculars P- -binoculars binoculars Now consider: - - PP- vs. - - PP-binoculars Alex Lascarides FNLP Lecture 13 6 Alex Lascarides FNLP Lecture 13 7

3 Practical issues Outline All this category-splitting makes the grammar much more specific (good!) But leads to huge grammar blowup and very sparse data (bad!) Lots of effort on how to balance these two issues. Complex smoothing schemes (similar to N-gram interpolation/backoff). More recently, increasing emphasis on automatically learned subcategories. 1. Dependencies: what/why 2. Transforming constituency dependency parse 3. Direct dependency parsing Transition-based (shift-reduce) Graph-based But do we really need phrase structure in the first place? Not always! Today: yntax (and parsing) out constituent structure. Alex Lascarides FNLP Lecture 13 8 Alex Lascarides FNLP Lecture 13 9 Lexicalized Constituency Parse V- - - PP- P remove the phrasal categories... Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 11

4 ... remove the (duplicated) terminals and collapse chains of duplicates... Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture and collapse chains of duplicates and collapse chains of duplicates... Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 15

5 ... and collapse chains of duplicates and collapse chains of duplicates... Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture and collapse chains of duplicates... Dependency Parse binoculars Linguists have long observed that the meanings of words in a sentence depend on one another, mostly in asymmetric, binary relations. Though some constructions don t cleanly fit this pattern: e.g., coordination, relative clauses. Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 19

6 Dependency Parse binoculars Equivalently, but showing word order (head modifier): Content vs. Functional Heads ome treebanks prefer content heads: Little were always watching Others prefer functional heads: Because it is a tree, every word has exactly one parent. Little were always watching Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Edge Labels It is often useful to distinguish different kinds of head modifier relations, by labeling edges: ROOT Dependency Paths For information extraction tasks involving real-world relationships between entities, chains of dependencies can provide good features: nsubj POBJ BJ DOBJ PREP amod prep pobj aux aux prep pobj poss amod Important relations for English include subject, direct object, determiner, adjective modifier, adverbial modifier, etc. (Different treebanks use somewhat different label sets.) British officials in Tehran have been meeting their Iranian counterparts How would you identify the subject in a constituency parse? (example from Brendan O Connor) Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 23

7 Projectivity Nonprojectivity A sentence s dependency parse is said to be projective if every subtree (node and all its descendants) occupies a contiguous span of the sentence. = The dependency parse can be drawn on top of the sentence out any crossing edges. ATT ATT BJ PC ATT ROOT A hearing on the issue is scheduled today VC TMP Other sentences are nonprojective: TMP ROOT ATT ATT BJ VC PC ATT A hearing is scheduled on the issue today Nonprojectivity is rare in English, but quite common in many languages. Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Dependencies: what/why Outline 2. Transforming constituency dependency parse 3. Direct dependency parsing Transition-based (shift-reduce) Graph-based Constituency Tree Dependency Tree We how the lexical head of the phrase can be used to collapse down to a dependency tree: V- - PP-binoculars P- -binoculars binoculars But how can we find each phrase s head in the first place? Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 27

8 Head Rules The standard solution is to use head rules: for every non-unary (P)CFG production, designate one RH nonterminal as containing the head., PP, PP P (content head), etc. V PP binoculars Heuristics to scale this to large grammars: e.g., in an, last immediate N child is the head. P Head Rules Then, propagate heads up the tree: - V- - P- PP -binoculars binoculars Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Head Rules Then, propagate heads up the tree: - - V- - P- PP -binoculars binoculars Head Rules Then, propagate heads up the tree: - - V- - PP-binoculars P- -binoculars binoculars Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 31

9 Head Rules Then, propagate heads up the tree: V- - PP-binoculars P- -binoculars binoculars Head Rules Then, propagate heads up the tree: V- - PP-binoculars P- -binoculars binoculars Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Dependencies: what/why Outline 2. Transforming constituency dependency parse 3. Direct dependency parsing Transition-based (shift-reduce) Graph-based Dependency Parsing ome of the algorithms you have seen for PCFGs can be adapted to dependency parsing. CKY can be adapted, though efficiency is a concern: obvious approach is O(Gn 5 ); Eisner algorithm brings it down to O(Gn 3 ) N. mith s slides explaining the Eisner algorithm: washington.edu/courses/cse517/16wi/slides/an-dep-slides.pdf hift-reduce: more efficient, doesn t even require a grammar! Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 35

10 Transitation-based Parsing: hift Reduce Parser Example tep tack Word List Action Relations 0 [root] [Kim,,andy] 1 [root,kim] [,andy] hift 2 [root,kim,] [andy] hift 3 [root,] [andy] LeftArc nsubj(,kim) 4 [root,,andy] [] hift 5 [root,] [] RightArc dobj(,andy) 6 [root] [] RightArc root book ROOT 3 possible actions: LeftArc: Assign head-dependent relation between s1 and s2; pop s2 RightArc: Assign head-dependent relation between s2 and s1; pop s1 hift: Put w1 on top of the stack. NUBJ DOBJ Kim andy Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Transition-based Parsing Latent structure is just edges between words. Train a classifier to predict next action (shift, reduce, attach-left, or attach-right), and proceed left-to-right through the sentence. O(n) time complexity! Only finds projective trees (out special extensions) Pioneering system: Nivre s MaltParser ee pdf (Jurafsky & Manning Coursera slides) for details and examples Graph-based Parsing Global algorithm: From the fully connected directed graph of all possible edges, choose the best ones that form a tree. Edge-factored models: Classifier assigns a nonnegative score to each possible edge; maximum spanning tree algorithm finds the spanning tree highest total score in O(n 2 ) time. Edge-factored assumption can be relaxed (higher-order models score larger units; more expensive). Unlabeled parse edge-labeling classifier (pipeline). Pioneering work: McDonald s MTParser Can be formulated as constraint-satisfaction integer linear programming (Martins s TurboParser) Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 39

11 Graph-based vs. Transition-based vs. Conversion-based TB: Features in scoring function can look at any part of the stack; no optimality guarantees for search; linear-time; (classically) projective only GB: Features in scoring function limited by factorization; optimal search in that model; quadratic-time; no projectivity constraint CB: In terms of accuracy, sometimes best to first constituency-parse, then convert to dependencies (e.g., tanford Parser). lower than direct methods. Choosing a Parser: Criteria Target representation: constituency or dependency? Efficiency? In practice, both runtime and memory use. Incrementality: parse the whole sentence at once, or obtain partial left-to-right analyses/expectations? Retrainable system? Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture Choosing a Parser: Performance OTA for English constituency parsing (WJ 23): 91% 92% F 1 Choosing a Parser: Performance Constituency parsing in other languages (Fernández-González and Martins, 2015) Alex Lascarides FNLP Lecture (Fernández-González and Martins, 2015) Alex Lascarides FNLP Lecture 13 43

12 Choosing a Parser: Performance OTA for English dependency parsing (WJ 23): 93% 94% UA, 91% 92% LA ummary While constituency parses give hierarchically nested phrases, dependency parses represent syntax trees whose edges connect words in the sentence. (No abstract phrase categories like.) Edges often labeled relations like subject. Head rules govern how a lexicalized constituency grammar can be extracted from a treebank, and how a constituency parse can be coverted to a dependency parse. For English, it is often fastest and most convenient to parse directly to dependencies. Two main paradigms, graph-based and transition-based, different kinds of models and search algorithms. (Zhou et al., 2015) Google online dependency parser. Try out the tanford parser and EMAFOR! Alex Lascarides FNLP Lecture Alex Lascarides FNLP Lecture 13 45