INCEpTION Workshop: Preference Learning

Transcription

1 INCEpTION Workshop: Preference Learning Outine of tak: Chaenges for efficient text annotation Introducing to preference earning Our preference earning approach: GPPL GPPL Experiments with argument convincingness Possibe appications to INCEpTION Sides by Edwin Simpson 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 1

2 Chaenges for efficient text annotation Discrete categories can be tricky - Exampe: sentiment cassification beyond binary poarity - I ove my mobie but woud not recommend it to any of my coeagues - Positive or negative? - Score from 1 to 5? Bue OR Negative OR Bad exampe...??? Green OR Positive OR Good exampe How can we make the annotator s task easier? - Speed up abeing - Reduce the number of annotations required How can we make annotation more consistent? - Over time - Between annotators 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 2

3 Introduction to Preference Learning Using preference earning to address the chaenges 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 3

4 Consider using Pairwise Comparisons When discrete categories and numerica ratings are tricky......ask the user to choose between two text items, for exampe: - Which phrase is more positive? - Which sentence provides a better dictionary exampe of the given word sense? X vs. Y Which is more bue? In genera: - Which item do you prefer? - Which item is more ike cass x? What can pairwise abes be used for? - To rank items - Choose good exampes of an annotation type - Annotate text without pre-defining discrete categories or numerica ratings 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 4

5 Benefits of Pairwise Preference Comparisons Lower cognitive oad on human annotators than categorising or rating (Kingsey, 2006; Kenda, 1948) - Quicker annotation during cod-start - Can annotate more text More consistent over time and between annotators: - Less ikey to change mind about x > y than whether x is rated 4 or 5 - Different annotators use ratings differenty, e.g. mainy using extreme or midding vaues, yet provide same answers to pairwise comparisons Aows fine-grained ranking of items - Fixed, pre-defined categories not required - Can focus on finding N best exampes - In cassification tasks, aows us to sort borderine exampes 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 5

6 Introducing Preference Learning Goa: earn to rank text items given pairwise preference abes - Automaticay find good exampes - Rank unseen data Where do we get pairwise abes? 1) Query user for pairwise comparisons of phrases/sentences/documents Train a mode to predict task-specific ranking over text items Minimise tota number of queries required 2) Learn from impicit feedback as users interact with the appication User seects a document from a ist = weak preference for that document Train mode to predict the most interesting documents for specific task Impicit preferences: - Seect the best transation in the ist - Seect the most positive/negative sentence - Find the most reevant document Expicit preferences: - Which transation is better - Which sentence is more positive/negative - Which diaogue utterance is more natura 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 6

7 Our Preference Learning Approach Inputs for training and prediction Set of text items, e.g. text spans, documents, sentences Inputs for training ony Pairwise preference abes Linguistic features Sentence embeddings GPPL: preference earning mode Each item has a atent preference score, which is a function of the item s features Preference scores for each item Item rankings Confidence in preference scores for each item Active earning Probabiities of pairwise preference abes Output predictions 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 7

8 The Latent Preference Function Maps text features to a preference score Items with simiar features have simiar scores Imagine text items have ony one rea-vaued feature: Item_1 < Item_2 Preference function Item_2 Item_1 feature We must earn this function from pairwise abes! 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 8

9 Use a Bayesian Approach to Hande Uncertainty How do we score items in areas with few observations? Gaussian process (GP) prior distribution on preference function - Confidence estimates (variance) [Chu & Ghahramani, 2005] - Mode compexity grows to fit the dataset - Buit-in reguarization through priors Preference function Items with many pairwise abes x x x +/- one standard deviation x x x x 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 9

10 Scaabe Approximate Bayesian Inference Compete mode is Gaussian process preference earning (GPPL) - GP priors on preference score function - Combined with the ikeihood mode for noisy pairwise abes Are Bayesian methods scaabe to reaistic dataset sizes? - Yes! They are now... - We deveoped a new inference method for GPPL - Stochastic variationa inference (SVI) [Hoffman et a., 2013] - Scaes sub-ineary in the number of pairwise abes and text items - Aows prediction over very arge corpora 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 10

11 Experiments Learning convincingness of arguments from pairwise judgements Datasets from: - I. Haberna & I. Gurevych, "Which argument is more convincing? Anayzing and predicting convincingness of Web arguments using bidirectiona LSTM," ACL, topics from two debate portas - Amazon Mechanica Turk Crowd abeed pairs of arguments - MACE: remove spam and produce singe abe per argument pair - Ranking god abes: PageRank Tasks: - Train on pairwise abes for 31 topics - Predict pairwise abes for test topic - Predict rankings for test topic 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 11

12 Resuts on Noisy, Crowdsourced Data Disagreement between workers eads to abe noise and conficting pairs, e.g. a<b, b<c, c<a Standard non- Bayesian approaches 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 12

13 Resuts on Noisy, Crowdsourced Data Strong overa performance with GPPL Bayesian cassifier aso performs we for predicting pairwise abes 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 13

14 Resuts: Scaabiity Runtime of GPPL with SVI not affected by number of text items (arguments) SVM, Bi-LSTM, and GPPL with aternative inference method increase poynomiay SVI: accuracy and runtime are a trade-off Adjust number of inducing points to change this trade-off We achieved good accuracy and runtimes with 500 inducing points 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 14

15 Resuts: Active Learning Simuated active earning - Iterativey sampe pairwise abes in batches of 2 - Predict pairwise cassifications for unseen pairs - In each iteration, choose the pair to be abeed using uncertainty samping - Labes chosen from previousycrowdsourced data GPPL earns more quicky with fewer abes Bi-LSTM accuracy decreases due to possibe overfitting 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 15

16 Appying Preference Learning to INCEpTION How can preference earning hep address the chaenges in text annotation? 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 16

17 Easier, More Consistent Annotation Use preference earning instead of numerica ratings Use preference earning instead of casses when some instances are ambiguous or mutipe casses coud appy Future work: combine cass abes or preference earning so that we use the easiest method for abeing each exampe Bue OR Negative OR Bad exampe...??? Green OR Positive OR Good exampe 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 17

18 Identifying Task-Specific Exampes from a Large Corpus Preference earning from impicit user feedback - When users seect documents/paragraphs to annotate, this generates pairwise abes Expicit user feedback: - Ask users to compare pairs of exampe texts to identify the best instances - Which document woud you prefer to spend time annotating? Future work: how to integrate pairwise comparisons into the user s workfow? vs. 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 18

19 Cod Start and Sma Data Probems Use Bayesian approaches with sma data Use active earning to speed up earning from cod-start Future work: identify the most effective active earning methods for specific use cases, e.g. finding the best N exampes Future work: transfer pre-trained modes from other tasks by setting priors System Performance Cod start Training Data Voume 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 19

20 Discussion How coud you imagine using preference earning? 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 20

21 Learning with Noisy Pairwise Labes If annotators change their minds or disagree, we get conficting pairs, e.g. a<b, b<c, c<a Difference between preference scores determines probabiity of pairwise abe p(item_1 < item_2 f(item_1), f(item_2)) = 0.8 Preference Item_2 score function Item_1 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 21

22 Infer the Preference Function Given the ikeihood function of the observed pairwise abes p(item_1 < item_2 f(item_1), f(item_2)) =......we can predict the preference scores for any text item Item_2...using probabiistic inference techniques - In practice, predictions are a function of the observed pairwise abes - Inference procedures estimate this function using optimisation or samping techniques - We choose a Bayesian inference method... 12/03/2018 Computer Science Department UKP Lab - Prof. Dr. Iryna Gurevych Edwin Simpson 22