Translation Quality Assessment: Evaluation and Estimation

Transcription

1 Translation Quality Assessment: Evaluation and Estimation Lucia Specia University of Sheffield MTM - Prague, 12 September 2016 Translation Quality Assessment: Evaluation and Estimation 1 / 38

2 Outline 1 Quality evaluation 2 Reference-based metrics 3 Quality estimation metrics 4 Metrics in the NMT era Translation Quality Assessment: Evaluation and Estimation 2 / 38

3 Outline 1 Quality evaluation 2 Reference-based metrics 3 Quality estimation metrics 4 Metrics in the NMT era Translation Quality Assessment: Evaluation and Estimation 3 / 38

4 Why do we care?... or why is this the first lecture of the Marathon? In the business of developing MT, we need to measure progress over new/alternative versions compare different MT systems decide whether a translation is good enough for something optimise parameters of MT systems understand where systems go wrong (diagnosis) Translation Quality Assessment: Evaluation and Estimation 4 / 38

5 Why do we care? One should optimise a system using the same metric that will be used to evaluate it Issue: how to choose a metric? Choice should be related to the purpose of the system will be used (not the case in practice) Other aspects are important for tuning (sentence/corpus-level, fast, cheap, differentiable,...) Translation Quality Assessment: Evaluation and Estimation 5 / 38

6 Complex problem MT evaluation is better understood than MT (Carbonell and Wilks, 1991) Translation Quality Assessment: Evaluation and Estimation 6 / 38

7 Complex problem MT evaluation is better understood than MT (Carbonell and Wilks, 1991) Translation Quality Assessment: Evaluation and Estimation 6 / 38

8 Complex problem MT evaluation is better understood than MT (Carbonell and Wilks, 1991) There are more MT evaluation metrics than MT approaches (Specia, 2016) Translation Quality Assessment: Evaluation and Estimation 6 / 38

9 Complex problem What does quality mean? Fluent? Adequate? Both? Easy to post-edit? System A better than system B?... Translation Quality Assessment: Evaluation and Estimation 7 / 38

10 Complex problem What does quality mean? Fluent? Adequate? Both? Easy to post-edit? System A better than system B?... Quality for whom/what? End-user (gisting vs dissemination) Post-editor (light vs heavy post-editing) Other applications (e.g. CLIR) MT-system (tuning or diagnosis for improvement)... Translation Quality Assessment: Evaluation and Estimation 7 / 38

11 Complex problem MT Do buy this product, it s their craziest invention! Translation Quality Assessment: Evaluation and Estimation 8 / 38

12 Complex problem MT Do buy this product, it s their craziest invention! HT Do not buy this product, it s their craziest invention! Translation Quality Assessment: Evaluation and Estimation 8 / 38

13 Complex problem MT Do buy this product, it s their craziest invention! HT Do not buy this product, it s their craziest invention! Severe if end-user does not speak source language Trivial to post-edit by translators Translation Quality Assessment: Evaluation and Estimation 8 / 38

14 Complex problem MT Six-hours battery, 30 minutes to full charge last. Translation Quality Assessment: Evaluation and Estimation 9 / 38

15 Complex problem MT Six-hours battery, 30 minutes to full charge last. HT The battery lasts 6 hours and it can be fully recharged in 30 minutes. Translation Quality Assessment: Evaluation and Estimation 9 / 38

16 Complex problem MT Six-hours battery, 30 minutes to full charge last. HT The battery lasts 6 hours and it can be fully recharged in 30 minutes. Ok for gisting - meaning preserved Very costly for post-editing if style is to be preserved Translation Quality Assessment: Evaluation and Estimation 9 / 38

17 A taxonomy of MT evaluation methods Manual Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

18 A taxonomy of MT evaluation methods Direct asses. Scoring Manual Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

19 A taxonomy of MT evaluation methods Is this translation correct? Translation Quality Assessment: Evaluation and Estimation 10 / 38

20 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

21 A taxonomy of MT evaluation methods Translation Quality Assessment: Evaluation and Estimation 10 / 38

22 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

23 A taxonomy of MT evaluation methods Translation Quality Assessment: Evaluation and Estimation 10 / 38

24 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

25 A taxonomy of MT evaluation methods HTER Translation Quality Assessment: Evaluation and Estimation 10 / 38

26 A taxonomy of MT evaluation methods Translation Quality Assessment: Evaluation and Estimation 10 / 38

27 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

28 A taxonomy of MT evaluation methods Translation Quality Assessment: Evaluation and Estimation 10 / 38

29 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Eye-tracking Automatic Translation Quality Assessment: Evaluation and Estimation 10 / 38

30 A taxonomy of MT evaluation methods Translation Quality Assessment: Evaluation and Estimation 10 / 38

31 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Eye-tracking Automatic Reference-based Translation Quality Assessment: Evaluation and Estimation 10 / 38

32 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Automatic Reference-based Quality estimation Eye-tracking BLEU, Meteor, NIST, TER, WER, PER, CDER, BEER, CiDER, Cobalt, RATATOUILLE, RED, AMBER, PARMESAN,... Translation Quality Assessment: Evaluation and Estimation 10 / 38

33 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Automatic Reference-based Quality estimation Eye-tracking BLEU, Meteor, NIST, TER, WER, PER, CDER, BEER, CiDER, Cobalt, RATATOUILLE, RED, AMBER, PARMESAN,... Translation Quality Assessment: Evaluation and Estimation 10 / 38

34 A taxonomy of MT evaluation methods Scoring Direct asses. Ranking Manual Error annotation Task-based Post-editing Reading comprehension Automatic Reference-based Quality estimation Eye-tracking BLEU, Meteor, NIST, TER, WER, PER, CDER, BEER, CiDER, Cobalt, RATATOUILLE, RED, AMBER, PARMESAN,... Translation Quality Assessment: Evaluation and Estimation 10 / 38

35 Outline 1 Quality evaluation 2 Reference-based metrics 3 Quality estimation metrics 4 Metrics in the NMT era Translation Quality Assessment: Evaluation and Estimation 11 / 38

36 Assumption The closer an MT system output is to a human translation (HT = reference), the better it is. Which system is better? MT 1 MT 2 Indignation in front of photos of a veiled woman controlled on the beach in Nice. Outrage at pictures of a veiled woman controlled on the beach in Nice. HT a Indignation at pictures of a veiled woman being checked on a beach in Nice. Translation Quality Assessment: Evaluation and Estimation 12 / 38

37 Assumption The closer an MT system output is to a human translation (HT = reference), the better it is. Which system is better? MT 1 MT 2 Indignation in front of photos of a veiled woman controlled on the beach in Nice. Outrage at pictures of a veiled woman controlled on the beach in Nice. HT a Indignation at pictures of a veiled woman being checked on a beach in Nice. Or, simply, how good is the MT 1 system output? Translation Quality Assessment: Evaluation and Estimation 12 / 38

38 Assumption Which system is better? MT 1 MT 2 Indignation in front of photos of a veiled woman controlled on the beach in Nice. Outrage at pictures of a veiled woman controlled on the beach in Nice. HT a HT b Indignation at pictures of a veiled woman being checked on a beach in Nice. Photos of a veiled woman checked by the police on the beach in Nice cause outrage. Translation Quality Assessment: Evaluation and Estimation 13 / 38

39 Assumption Which system is better? MT 1 MT 2 Indignation in front of photos of a veiled woman controlled on the beach in Nice. Outrage at pictures of a veiled woman controlled on the beach in Nice. HT a HT b Indignation at pictures of a veiled woman being checked on a beach in Nice. Photos of a veiled woman checked by the police on the beach in Nice cause outrage. Or, again, how good is the MT 1 system output? Translation Quality Assessment: Evaluation and Estimation 13 / 38

40 BLEU BLEU: BiLingual Evaluation Understudy Most widely used metric, both for MT system evaluation/comparison and SMT tuning Matching of n-grams between MT and HT: rewards same words in equal order #clip(g) count of reference n-grams g which happen in a MT sentence h clipped by the number of times g appears in the HT sentence for h; #(g ) = number of n-grams in MT output n-gram precision p n for a set of translations in C: p n = c C c C g ngrams(c) #clip(g) g ngrams(c) #(g ) Translation Quality Assessment: Evaluation and Estimation 14 / 38

41 BLEU Combine (mean of the log) 1-n n-gram precisions log p n n Translation Quality Assessment: Evaluation and Estimation 15 / 38

42 BLEU Combine (mean of the log) 1-n n-gram precisions log p n n Bias towards translations with fewer words Brevity penalty to penalise MT sentences that are shorter than reference Compares the overall number of words w h of the entire hypotheses set with { ref length w r : 1 if w c w r BP = (1 wr /wc) e otherwise Translation Quality Assessment: Evaluation and Estimation 15 / 38

43 BLEU Combine (mean of the log) 1-n n-gram precisions log p n n Bias towards translations with fewer words Brevity penalty to penalise MT sentences that are shorter than reference Compares the overall number of words w h of the entire hypotheses set with { ref length w r : 1 if w c w r BP = (1 wr /wc) e otherwise ( ) BLEU = BP exp log p n Translation Quality Assessment: Evaluation and Estimation 15 / 38 n

44 BLEU Scale: 0-1, but highly dependent on the test set Rewards fluency by matching high n-grams (up to 4) Rewards adequacy by unigrams and brevity penalty poor model of recall Synonyms and paraphrases only handled if in one of reference translations All tokens are equally weighted: incorrect content word = incorrect determiner Translation Quality Assessment: Evaluation and Estimation 16 / 38

45 BLEU Scale: 0-1, but highly dependent on the test set Rewards fluency by matching high n-grams (up to 4) Rewards adequacy by unigrams and brevity penalty poor model of recall Synonyms and paraphrases only handled if in one of reference translations All tokens are equally weighted: incorrect content word = incorrect determiner Better for evaluating changes in the same system than comparing different MT architectures Translation Quality Assessment: Evaluation and Estimation 16 / 38

46 BLEU Example: MT: in two weeks Iraq s weapons will give army HT: the Iraqi weapons are to be handed over to the army within two weeks 1-gram precision: 4/8 2-gram precision: 1/7 3-gram precision: 0/6 4-gram precision: 0/5 Translation Quality Assessment: Evaluation and Estimation 17 / 38

47 Edit distance metrics TER: Translation Error Rate Levenshtein edit distance Minimum proportion of insertions, deletions, and substitutions to transform MT sentence into HT Adds shift operation Translation Quality Assessment: Evaluation and Estimation 18 / 38

48 Edit distance metrics TER: Translation Error Rate Levenshtein edit distance Minimum proportion of insertions, deletions, and substitutions to transform MT sentence into HT Adds shift operation REF: SAUDI ARABIA denied this week information published in the AMERICAN new york times HYP: [this week] the saudis denied information published in the ***** new york times Translation Quality Assessment: Evaluation and Estimation 18 / 38

49 Edit distance metrics TER: Translation Error Rate Levenshtein edit distance Minimum proportion of insertions, deletions, and substitutions to transform MT sentence into HT Adds shift operation REF: SAUDI ARABIA denied this week information published in the AMERICAN new york times HYP: [this week] the saudis denied information published in the ***** new york times 1 shift, 2 substit., 1 deletion: TER = 4 13 = 0.31 Translation Quality Assessment: Evaluation and Estimation 18 / 38

50 Edit distance metrics TER: Translation Error Rate Levenshtein edit distance Minimum proportion of insertions, deletions, and substitutions to transform MT sentence into HT Adds shift operation REF: SAUDI ARABIA denied this week information published in the AMERICAN new york times HYP: [this week] the saudis denied information published in the ***** new york times 1 shift, 2 substit., 1 deletion: TER = 4 13 = 0.31 Human-targeted TER (HTER) TER between MT and its post-edited version Translation Quality Assessment: Evaluation and Estimation 18 / 38

51 Alignment-based metrics METEOR: Unigram Precision and Recall Align MT & HT Matching considers inflection variants (stems), synonyms, paraphrases Fluency addressed via a direct penalty: fragmentation of the matching METEOR score = F-mean score discounted for fragmentation = F-mean * (1 - DF) Parameters can be trained Translation Quality Assessment: Evaluation and Estimation 19 / 38

52 Alignment-based metrics MT: in two weeks Iraq s weapons will give army HT: the Iraqi weapons are to be handed over to the army within two weeks Translation Quality Assessment: Evaluation and Estimation 20 / 38

53 Alignment-based metrics MT: in two weeks Iraq s weapons will give army HT: the Iraqi weapons are to be handed over to the army within two weeks Matching: MT two weeks Iraq s weapons army HT: Iraqi weapons army two weeks Translation Quality Assessment: Evaluation and Estimation 20 / 38

54 Alignment-based metrics MT: in two weeks Iraq s weapons will give army HT: the Iraqi weapons are to be handed over to the army within two weeks Matching: MT two weeks Iraq s weapons army HT: Iraqi weapons army two weeks P = 5/8 =0.625 R = 5/14 = F-mean = 10*P*R/(9P+R) = Translation Quality Assessment: Evaluation and Estimation 20 / 38

55 Alignment-based metrics MT: in two weeks Iraq s weapons will give army HT: the Iraqi weapons are to be handed over to the army within two weeks Matching: MT two weeks Iraq s weapons army HT: Iraqi weapons army two weeks P = 5/8 =0.625 R = 5/14 = F-mean = 10*P*R/(9P+R) = Fragmentation: 3 frags for 5 words = (3)/(5) = 0.6 Discounting factor: DF = 0.5 * (0.6**3) = METEOR: F-mean * (1 - DF) = * = Translation Quality Assessment: Evaluation and Estimation 20 / 38

56 BEER BEER: BEtter Evaluation as Ranking Trained metric score(h, r) = i w i φ i (h, r) = w φ Learns from pairwise rankings Various features between MT output and reference translation Precision, Recall and F1 over character n-grams (1-6) Idem for word unigrams: content vs function separately Reordering through permutation trees and distance to ideal monotone permutation Translation Quality Assessment: Evaluation and Estimation 21 / 38

57 Dozens more... Some - WMT metrics task: CharacTer chrf/wordf TerroCat MEANT and TINE TESLA LEPOR ROSE AMBER Many other linguistically motivated metrics where matching goes beyond word forms... Asiya toolkit - up until 2014 Translation Quality Assessment: Evaluation and Estimation 22 / 38

58 Dozens more... WMT16 metrics task (by Bojar et al.): Translation Quality Assessment: Evaluation and Estimation 23 / 38

59 Problems with reference-based evaluation Reference(s): subset of good translations, usually one Some metrics expand matching, e.g. synonyms in Meteor Huge variation in reference translations. E.g. Source 不过这一切都由不得你 However these all totally beyond the control of you. MT But all this is beyond the control of you. Human score BLEU score HT 1 But all this is beyond your control HT 2 However, you cannot choose yourself HT 3 However, not everything is up to you to decide HT 4 But you can t choose that Metrics completely disregard source segment Cannot be applied for MT systems in use Translation Quality Assessment: Evaluation and Estimation 24 / 38

60 Problems with reference-based evaluation Reference(s): subset of good translations, usually one Some metrics expand matching, e.g. synonyms in Meteor Huge variation in reference translations. E.g. Source 不过这一切都由不得你 However these all totally beyond the control of you. MT But all this is beyond the control of you. Human score BLEU score HT 1 But all this is beyond your control HT 2 However, you cannot choose yourself HT 3 However, not everything is up to you to decide HT 4 But you can t choose that Metrics completely disregard source segment Cannot be applied for MT systems in use Translation Quality Assessment: Evaluation and Estimation 24 / 38

61 Problems with reference-based evaluation Reference(s): subset of good translations, usually one Some metrics expand matching, e.g. synonyms in Meteor Huge variation in reference translations. E.g. Source 不过这一切都由不得你 However these all totally beyond the control of you. MT But all this is beyond the control of you. Human score BLEU score HT 1 But all this is beyond your control HT 2 However, you cannot choose yourself HT 3 However, not everything is up to you to decide HT 4 But you can t choose that Metrics completely disregard source segment Cannot be applied for MT systems in use Translation Quality Assessment: Evaluation and Estimation 24 / 38

62 Outline 1 Quality evaluation 2 Reference-based metrics 3 Quality estimation metrics 4 Metrics in the NMT era Translation Quality Assessment: Evaluation and Estimation 25 / 38

63 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Translation Quality Assessment: Evaluation and Estimation 26 / 38

64 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Quality defined by the data: purpose is clear, no comparison to references, source considered Translation Quality Assessment: Evaluation and Estimation 26 / 38

65 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Quality defined by the data: purpose is clear, no comparison to references, source considered Quality = Can we publish it as is? Translation Quality Assessment: Evaluation and Estimation 26 / 38

66 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Quality defined by the data: purpose is clear, no comparison to references, source considered Quality = Can we publish it as is? Quality = Can a reader get the gist? Translation Quality Assessment: Evaluation and Estimation 26 / 38

67 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Quality defined by the data: purpose is clear, no comparison to references, source considered Quality = Can we publish it as is? Quality = Can a reader get the gist? Quality = Is it worth post-editing it? Translation Quality Assessment: Evaluation and Estimation 26 / 38

68 QE - Overview Quality estimation (QE): metrics that provide an estimate on the quality of translations on the fly Quality defined by the data: purpose is clear, no comparison to references, source considered Quality = Can we publish it as is? Quality = Can a reader get the gist? Quality = Is it worth post-editing it? Quality = How much effort to fix it? Translation Quality Assessment: Evaluation and Estimation 26 / 38

69 QE - Framework Building a model: X: examples of source & translations Feature extraction Features Y: Quality scores for examples in X Machine Learning QE model Translation Quality Assessment: Evaluation and Estimation 27 / 38

70 QE - Framework Applying the model: MT system Source Text x s' Translation for x t' Feature extraction Features Quality score y' QE model Translation Quality Assessment: Evaluation and Estimation 27 / 38

71 Data and levels of granularity Sentence level: 1-5 subjective scores, PE time, PE edits Word level: good/bad, good/delete/replace, MQM Phrase level: good/bad Document level: PE effort Translation Quality Assessment: Evaluation and Estimation 28 / 38

72 Features and algorithms Adequacy features Source text MT system Translation s -1 s s +1 t -1 t t +1 Complexity features Confidence features Fluency features Algorithms can be used off-the-shelf Translation Quality Assessment: Evaluation and Estimation 29 / 38

73 QE - baseline setting Features: number of tokens in the source and target sentences average source token length average number of occurrences of words in the target number of punctuation marks in source and target sentences LM probability of source and target sentences average number of translations per source word % of seen source n-grams Translation Quality Assessment: Evaluation and Estimation 30 / 38

74 QE - baseline setting Features: number of tokens in the source and target sentences average source token length average number of occurrences of words in the target number of punctuation marks in source and target sentences LM probability of source and target sentences average number of translations per source word % of seen source n-grams SVM regression with RBF kernel Translation Quality Assessment: Evaluation and Estimation 30 / 38

75 QE - baseline setting Features: number of tokens in the source and target sentences average source token length average number of occurrences of words in the target number of punctuation marks in source and target sentences LM probability of source and target sentences average number of translations per source word % of seen source n-grams SVM regression with RBF kernel QuEst: Translation Quality Assessment: Evaluation and Estimation 30 / 38

76 QE - SoA sentence-level Predicting HTER (WMT16) System ID Pearson Spearman English-German YSDA/SNTX+BLEU+SVM POSTECH/SENT-RNN-QV SHEF-LIUM/SVM-NN-emb-QuEst POSTECH/SENT-RNN-QV SHEF-LIUM/SVM-NN-both-emb UGENT-LT3/SCATE-SVM UFAL/MULTIVEC RTM/RTM-FS-SVR UU/UU-SVM UGENT-LT3/SCATE-SVM RTM/RTM-SVR Baseline SVM SHEF/SimpleNets-SRC SHEF/SimpleNets-TGT Translation Quality Assessment: Evaluation and Estimation 31 / 38

77 Outline 1 Quality evaluation 2 Reference-based metrics 3 Quality estimation metrics 4 Metrics in the NMT era Translation Quality Assessment: Evaluation and Estimation 32 / 38

78 SMT vs NMT Pearson correlation with DA scores for popular metrics on 200 sentences from WMT16 s uedin SMT and NMT systems: uedin-pbmt uedin-nmt BLEU Meteor TER chrf BEER UPF-Cobalt CobaltF-comp DPMFcomb (Work with Marina Fomicheva) Translation Quality Assessment: Evaluation and Estimation 33 / 38

79 Are metrics better for NMT because systems are better? Correlation with DA scores on 840 low-quality (Q1-human) & 840 high-quality (Q4-human) sentences (all systems) Q1 - low quality Q4 - high quality BLEU Meteor TER UPF-Cobalt CobaltF-comp DPMFcomb BEER chrf (Work with Marina Fomicheva) Translation Quality Assessment: Evaluation and Estimation 34 / 38

80 Or was it a feature of the uedin systems? Correlation of various MT systems on 400 sentences per group: PBMT PBMT + NMT Syntax BLEU Meteor TER chrf BEER UPF-Cobalt CobaltF-comp MetricsF DPMFcomb These NMT systems only use neural models for rescoring. Also, average DA scores not higher for the PMT+NMT group (Work with Marina Fomicheva) Translation Quality Assessment: Evaluation and Estimation 35 / 38

81 Conclusions (Machine) Translation evaluation is still an open problem Translation Quality Assessment: Evaluation and Estimation 36 / 38

82 Conclusions (Machine) Translation evaluation is still an open problem Quality estimation and other trained metrics can learn different versions of quality Translation Quality Assessment: Evaluation and Estimation 36 / 38

83 Conclusions (Machine) Translation evaluation is still an open problem Quality estimation and other trained metrics can learn different versions of quality Which metrics are used in practice? BLEU + your favourite other And same metric for tuning Translation Quality Assessment: Evaluation and Estimation 36 / 38

84 Conclusions (Machine) Translation evaluation is still an open problem Quality estimation and other trained metrics can learn different versions of quality Which metrics are used in practice? BLEU + your favourite other And same metric for tuning And for official comparisons? WMT: manual ranking and direct assessment IWSLT: manual post-editing Translation Quality Assessment: Evaluation and Estimation 36 / 38

85 Conclusions (Machine) Translation evaluation is still an open problem Quality estimation and other trained metrics can learn different versions of quality Which metrics are used in practice? BLEU + your favourite other And same metric for tuning And for official comparisons? WMT: manual ranking and direct assessment IWSLT: manual post-editing Are our metrics good at assessing NMT systems? Are these metrics good to optimise NMT systems? Translation Quality Assessment: Evaluation and Estimation 36 / 38

86 Translation Quality Assessment: Evaluation and Estimation Lucia Specia University of Sheffield MTM - Prague, 12 September 2016 Translation Quality Assessment: Evaluation and Estimation 37 / 38

87 Conclusions MT system Type Average score Segments AFRL-MITLL-Phrase PBMT + NMT AFRL-MITLL-contrast PBMT + NMT AMU-UEDIN PBMT + NMT KIT PBMT + NMT LIMSI PBMT NRC PBMT PJATK PBMT PROMT-Rule-based RBMT PROMT-SMT PBMT UH-factored PBMT UH-opus PBMT cu-mergedtrees Syntax PBMT dvorkanton PBMT + NMT jhu-pbmt PBMT jhu-syntax Syntax PBMT online-b PBMT online-f PBMT online-g PBMT tbtk-syscomb PBMT uedin-nmt NMT uedin-pbmt PBMT uedin-syntax Syntax PBMT Translation Quality Assessment: Evaluation and Estimation 38 / 38