Phrase-level Quality Estimation for Machine Translation

Transcription

1 Phrase-level Quality Estimation for Machine Translation Varvara Logacheva, Lucia Specia University of Sheffield December 3, 2015 Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 1/21

2 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

3 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

4 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog OK OK OK BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

5 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog OK OK OK BAD Sentence-level quality estimation: C est mon chat. My dog likes chocolate. This is my cat. Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

6 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog OK OK OK BAD Sentence-level quality estimation: C est mon chat. My dog likes chocolate. This is my cat. BAD OK Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

7 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog OK OK OK BAD Sentence-level quality estimation: C est mon chat. Document-level quality estimation: Il a besoin de toilettage régulier car le poil du Maine Coon est un poil, mi-long, il ne faut pas être allergique! Il a aussi besoin d un, shampoing tout les mois, et d un dégraissant tout les deux mois (ou, avant les expositions). Il existe d ailleurs des shampoings pour, sublimer les couleurs (comme le blanc ou le noir par exemple). My dog likes chocolate. This is my cat. BAD OK He needs regular grooming for the coat of Maine Coon is a semi-long hair, it should not be allergic! He also needs a shampoo every month, and a degreasing agent every two months (or before exposure). There are also shampoos to sublimate the colors (like white or black for example). Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

8 Quality Estimation Definition Phrase-level: Motivation Phrase-level: Challenges Quality Estimation determination of the quality of an automatically translated segment without reference translation. Word-level quality estimation: C est mon chat. This is my dog OK OK OK BAD Sentence-level quality estimation: C est mon chat. Document-level quality estimation: Il a besoin de toilettage régulier car le poil du Maine Coon est un poil, mi-long, il ne faut pas être allergique! Il a aussi besoin d un, shampoing tout les mois, et d un dégraissant tout les deux mois (ou, avant les expositions). Il existe d ailleurs des shampoings pour, sublimer les couleurs (comme le blanc ou le noir par exemple). My dog likes chocolate. This is my cat. BAD OK He needs regular grooming for the coat of Maine Coon is a semi-long hair, it should not be allergic! He also needs a shampoo every month, and a degreasing agent every two months (or before exposure). There are also shampoos to sublimate the colors (like white or black for example). OK Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 2/21

9 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 3/21

10 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Source: Machine Translation: A beautiful flower Un bel arbre Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 4/21

11 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Source: Machine Translation: Post-edition: A beautiful flower Un bel arbre Une belle fleur 3 errors Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 4/21

12 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Machine Translation: Post-edition: Un bel arbre Une belle fleur Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 5/21

13 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Machine Translation: Un bel arbre Wrong Post-edition: Une belle fleur choice of word Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 5/21

14 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Machine Translation: Un bel arbre Wrong Post-edition: Une belle fleur agreement Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 5/21

15 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges Machine Translation errors are not independent Machine Translation: Post-edition: Un bel arbre Une belle fleur 1 phrase-level error Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 5/21

16 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges The majority of Machine Translation systems are phrase-level Morgen fliege ich nach Kanada zur Konferenz Tomorrow I will fly to the party in Canada Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 6/21

17 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges The majority of Machine Translation systems are phrase-level Morgen fliege ich nach Kanada zur Konferenz Tomorrow I will fly to the party in Canada Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 6/21

18 Phrase-level QE: Motivation Definition Phrase-level: Motivation Phrase-level: Challenges The majority of Machine Translation systems are phrase-level Morgen fliege ich nach Kanada zur Konferenz Tomorrow I will fly to the party in Canada BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 6/21

19 Phrase-level QE: Challenges Definition Phrase-level: Motivation Phrase-level: Challenges Problems: No information on phrase borders No definition of phrase All labels are word-level Optimal features for phrase-level QE are unknown Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 7/21

20 Phrase-level QE: Challenges Definition Phrase-level: Motivation Phrase-level: Challenges Problems: No information on phrase borders No definition of phrase All labels are word-level Optimal features for phrase-level QE are unknown Sub-tasks: Segment sentences into phrases Retrieve phrase-level labels Find well-performing phrase-level features Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 7/21

21 Data segmentation Quality Estimation Source-target Target Labelling Available data: Source sentence Automatically translated sentence Post-edition of automatic translation Word-level labelling of automatic translation (OK/BAD) Approach: re-decoding of the data. Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 8/21

22 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

23 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Train phrase table on the QE data Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

24 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Train phrase table on the QE data Decode source side so that the output matches the target side Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

25 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Train phrase table on the QE data Decode source side so that the output matches the target side Decoder returns phrase segmentation: Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

26 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Train phrase table on the QE data Decode source side so that the output matches the target side Decoder returns phrase segmentation: Source: Well, things got even more inappropriate! Target: Bueno, las cosas se pusieron aún más inadecuado! Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

27 Source-target Target Labelling Decoder-based segmentation: source-target Joint segmentation of source and target sentences with constrained decoding Train phrase table on the QE data Decode source side so that the output matches the target side Decoder returns phrase segmentation: Source: Well, things got even more inappropriate! Target: Bueno, las cosas se pusieron aún más inadecuado! Pros Correspondence between source and target phrases Contras Phrases are too long Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 9/21

28 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

29 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

30 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

31 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Align source and target sides Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

32 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Align source and target sides Well, things got even more inappropriate! Bueno, las cosas se pusieron aún más inadecuado! Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

33 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Align source and target sides Well, things got even more inappropriate! Bueno, las cosas se pusieron aún más inadecuado! Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

34 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Align source and target sides Well, things got even more inappropriate! Bueno, las cosas se pusieron aún más inadecuado! Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

35 Source-target Target Labelling Decoder-based segmentation: target Independent decoding of target side Translate target side with target-to-source SMT system Keep phrase segmentation Align source and target sides Well, things got even more inappropriate! Bueno, las cosas se pusieron aún más inadecuado! Pros Shorter phrases Contras Unreliable source phrases: depend on alignments don t guarantee source coverage Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 10/21

36 Labelling Quality Estimation Source-target Target Labelling Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 11/21

37 Labelling Quality Estimation Source-target Target Labelling Optimistic: majority labelling [ Mostly good phrase ] [ mostly bad phrase ] OK OK BAD OK BAD BAD OK BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 11/21

38 Labelling Quality Estimation Source-target Target Labelling Optimistic: majority labelling [ Mostly good phrase ] [ mostly bad phrase ] OK OK BAD OK BAD BAD OK BAD Pessimistic: 30% bad words [ Bad enough phrase ] [ not enough bad phrase ] OK OK BAD OK OK OK BAD BAD OK Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 11/21

39 Labelling Quality Estimation Source-target Target Labelling Optimistic: majority labelling [ Mostly good phrase ] [ mostly bad phrase ] OK OK BAD OK BAD BAD OK BAD Pessimistic: 30% bad words [ Bad enough phrase ] [ not enough bad phrase ] OK OK BAD OK OK OK BAD BAD OK Super-pessimistic: any phrase with a bad word is bad [ Good phrase ] [ bad phrase ] [ yet another bad phrase ] OK OK BAD OK OK OK BAD OK OK BAD BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 11/21

40 Experimental setup Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 12/21

41 Experimental setup Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Language pair: English Spanish Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 12/21

42 Experimental setup Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Language pair: English Spanish Datasets: WMT-14: 2,000 sentences, manual labelling WMT-15: 11,000 sentences, post-edited Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 12/21

43 Experimental setup Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Language pair: English Spanish Datasets: WMT-14: 2,000 sentences, manual labelling WMT-15: 11,000 sentences, post-edited Feature sets: QuEst sentence-level features Word2Vec word embeddings QuEst + Word2Vec features Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 12/21

44 Experimental setup Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Language pair: English Spanish Datasets: WMT-14: 2,000 sentences, manual labelling WMT-15: 11,000 sentences, post-edited Feature sets: QuEst sentence-level features Word2Vec word embeddings QuEst + Word2Vec features Training methods: Conditional Random Fields Random Forest Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 12/21

45 properties Experimental setup Phrase-level data properties Parameters selection Results 100% 80% 60% 40% 20% 0% Target Source-Target 1 word 2 words 3 words 4 words 5 words Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 13/21

46 "BAD" to "OK", % Labelling properties Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results optimistic pessimistic super-pessimistic source-target target OK" to "BAD", % Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 14/21

47 F1 OK Quality Estimation Optimal parameters: labelling Experimental setup Phrase-level data properties Parameters selection Results 1 0,8 0,6 0,4 0,2 Pessimistic Super-pessimistic 0 0 0,2 0,4 0,6 F1 BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 15/21

48 F1 OK Quality Estimation Optimal parameters: features Experimental setup Phrase-level data properties Parameters selection Results 1 0,8 0,6 0,4 0,2 0 QuEst features Word2Vec features QuEst+Word2Vec features 0 0,2 0,4 0,6 F1 BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 16/21

49 F1 OK Quality Estimation Experimental setup Phrase-level data properties Parameters selection Results Optimal parameters: segmentation and training algorithms 1 0,8 0,6 0,4 0,2 0 Source Target Classification Sequence labelling Best result 0 0,1 0,2 0,3 0,4 0,5 0,6 F1 BAD Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 17/21

50 Comparison with word-level systems Experimental setup Phrase-level data properties Parameters selection Results WMT-14 systems System F 1 -BAD F 1 -OK Weighted F1 phrase-wmt Baseline-all-bad FBK-UPV-UEDIN LIG Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 18/21

51 Comparison with word-level systems Experimental setup Phrase-level data properties Parameters selection Results WMT-15 systems System F 1 -BAD F 1 -OK Weighted F1 phrase-wmt UAlacant SHEF-word2vec Baseline-all-bad Baseline Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 19/21

52 Quality Estimation phrase-level QE model for MT Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 20/21

53 Quality Estimation phrase-level QE model for MT decoder-based sentence segmentation techniques Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 20/21

54 Quality Estimation phrase-level QE model for MT decoder-based sentence segmentation techniques features: sentence-level and word embeddings Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 20/21

55 Quality Estimation phrase-level QE model for MT decoder-based sentence segmentation techniques features: sentence-level and word embeddings training methods: CRF and Random Forest Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 20/21

56 Quality Estimation phrase-level QE model for MT decoder-based sentence segmentation techniques features: sentence-level and word embeddings training methods: CRF and Random Forest phrase-level systems outperform word-level systems Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 20/21

57 Thank you Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 21/21