Translation Quality Assessment: Evaluation and Estimation
|
|
|
- Emily Foster
- 6 years ago
- Views:
Transcription
1 Translation Quality Assessment: Evaluation and Estimation Lucia Specia University of Sheffield 9 September 2013 Translation Quality Assessment: Evaluation and Estimation 1 / 33
2 Overview MT Evaluation is better understood than MT (Carbonell and Wilks, 1991) Translation Quality Assessment: Evaluation and Estimation 2 / 33
3 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 3 / 33
4 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 4 / 33
5 Overview What does quality mean? Fluent? Adequate? Easy to post-edit? Translation Quality Assessment: Evaluation and Estimation 5 / 33
6 Overview What does quality mean? Fluent? Adequate? Easy to post-edit? Quality for whom? End-user MT-system (tuning) Post-editor Other applications (e.g. CLIR) Translation Quality Assessment: Evaluation and Estimation 5 / 33
7 Overview What does quality mean? Fluent? Adequate? Easy to post-edit? Quality for whom? End-user MT-system (tuning) Post-editor Other applications (e.g. CLIR) Quality for what? Internal communications Dissemination (publishing) Gisting (Google Translate) Draft translations (light vs heavy post-editing) MT system improvement (diagnosis) Translation Quality Assessment: Evaluation and Estimation 5 / 33
8 Overview ref Do not buy this product, it s their craziest invention! sys Do buy this product, it s their craziest invention! Translation Quality Assessment: Evaluation and Estimation 6 / 33
9 Overview ref Do not buy this product, it s their craziest invention! sys Do buy this product, it s their craziest invention! Severe if end-user does not speak source language Trivial to post-edit by translators Translation Quality Assessment: Evaluation and Estimation 6 / 33
10 Overview ref Do not buy this product, it s their craziest invention! sys Do buy this product, it s their craziest invention! Severe if end-user does not speak source language Trivial to post-edit by translators ref The battery lasts 6 hours and it can be fully recharged in 30 minutes. sys Six-hours battery, 30 minutes to full charge last. Translation Quality Assessment: Evaluation and Estimation 6 / 33
11 Overview ref Do not buy this product, it s their craziest invention! sys Do buy this product, it s their craziest invention! Severe if end-user does not speak source language Trivial to post-edit by translators ref The battery lasts 6 hours and it can be fully recharged in 30 minutes. sys Six-hours battery, 30 minutes to full charge last. Ok for gisting - meaning preserved Very costly for post-editing if style is to be preserved Translation Quality Assessment: Evaluation and Estimation 6 / 33
12 Overview How do we measure quality? Human metrics: error counts (which?), ranking, acceptability, 1-N fluency/adequacy Automatic metrics based on human references: (BLEU, METEOR, TER, etc. Semi-automatic metrics based on post-editions: HTER, PE time, eye-tracking, etc. Translation Quality Assessment: Evaluation and Estimation 7 / 33
13 Overview How do we measure quality? Human metrics: error counts (which?), ranking, acceptability, 1-N fluency/adequacy Automatic metrics based on human references: (BLEU, METEOR, TER, etc. Semi-automatic metrics based on post-editions: HTER, PE time, eye-tracking, etc. Automatic metrics without references: quality estimation Translation Quality Assessment: Evaluation and Estimation 7 / 33
14 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 8 / 33
15 QTLaunchPad project Multidimensional Quality Metrics (MQM) based on a specification Machine and human translation quality Manual and (semi-)automatic assessment Takes quality of source text into account MT system improvement, gisting, dissemination, etc. Translation Quality Assessment: Evaluation and Estimation 9 / 33
16 Multidimensional Quality Metrics (MQM) Issues selected based on a given specification (dimensions): Language/locale Subject field/domain Terminology Text Type Audience Purpose Register Style Content correspondence Output modality,... Translation Quality Assessment: Evaluation and Estimation 10 / 33
17 Multidimensional Quality Metrics (MQM) Issue types (core): Translation Quality Assessment: Evaluation and Estimation 11 / 33
18 Multidimensional Quality Metrics (MQM) Issue types: high-level-structure-0 Combining issue types: TQ = 100 AccP (FluP T FluP S ) (VerP T VerP S ) Translation Quality Assessment: Evaluation and Estimation 12 / 33
19 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 13 / 33
20 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Translation Quality Assessment: Evaluation and Estimation 14 / 33
21 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Translation Quality Assessment: Evaluation and Estimation 14 / 33
22 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Quality = Can we publish it as is? Translation Quality Assessment: Evaluation and Estimation 14 / 33
23 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Quality = Can we publish it as is? Quality = Can a reader get the gist? Translation Quality Assessment: Evaluation and Estimation 14 / 33
24 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Quality = Can we publish it as is? Quality = Can a reader get the gist? Quality = Is it worth post-editing it? Translation Quality Assessment: Evaluation and Estimation 14 / 33
25 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Quality = Can we publish it as is? Quality = Can a reader get the gist? Quality = Is it worth post-editing it? Quality = How much effort to fix it? Translation Quality Assessment: Evaluation and Estimation 14 / 33
26 Overview Quality estimation (QE): metrics that provide an estimate on the quality of unseen translations Quality defined by the data Quality = Can we publish it as is? Quality = Can a reader get the gist? Quality = Is it worth post-editing it? Quality = How much effort to fix it? Quality = What s this translation s MQM score? Translation Quality Assessment: Evaluation and Estimation 14 / 33
27 Framework X: examples of source & translations Feature extraction Features Y: Quality scores for examples in X Machine Learning QE model Translation Quality Assessment: Evaluation and Estimation 15 / 33
28 Framework MT system Source Text x s' Translation for x t' Feature extraction Features Quality score y' QE model Translation Quality Assessment: Evaluation and Estimation 15 / 33
29 Framework Main components to build a QE system: 1 Definition of quality: what to predict 2 (Human) labelled data (for quality) 3 Features 4 Machine learning algorithm Translation Quality Assessment: Evaluation and Estimation 16 / 33
30 Definition of quality Predict 1-N absolute scores for adequacy/fluency Predict 1-N absolute scores for post-editing effort Predict average post-editing time per word Predict relative rankings Predict relative rankings for same source Predict percentage of edits needed for sentence Predict word-level edits and its types Predict BLEU, etc. scores for document Translation Quality Assessment: Evaluation and Estimation 17 / 33
31 Datasets SHEF (several): people/l.specia/resources.html LIG (10K, fr-en): User/marion.potet/index.php?page=download LMSI (14K, fr-en, en-fr, 2 post-editors): recherche/index.html Translation Quality Assessment: Evaluation and Estimation 18 / 33
32 Features Adequacy indicators Source text MT system Translation Complexity indicators Confidence indicators Fluency indicators Translation Quality Assessment: Evaluation and Estimation 19 / 33
33 QuEst Goal: framework to explore features for QE Feature extractors for 150+ features of all types: Java Machine learning: Gaussian Processes & scikit-learn toolkit (Python), with wrappers for a number of algorithms, grid search, feature selection Open source: Translation Quality Assessment: Evaluation and Estimation 20 / 33
34 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 21 / 33
35 Shared Task WMT12-13 with Radu Soricut & Christian Buck Translation Quality Assessment: Evaluation and Estimation 22 / 33
36 Shared Task WMT12-13 with Radu Soricut & Christian Buck Sentence- and word-level estimation of PE effort Translation Quality Assessment: Evaluation and Estimation 22 / 33
37 Shared Task WMT12-13 with Radu Soricut & Christian Buck Sentence- and word-level estimation of PE effort Datasets and language pairs: Quality Year Languages 1-5 subjective scores WMT12 en-es Ranking all sentences best-worst WMT12/13 en-es HTER scores WMT13 en-es Post-editing time WMT13 en-es Word-level edits: change/keep WMT13 en-es Word-level edits: keep/delete/replace WMT13 en-es Ranking 5 MTs per source WMT13 en-es; de-en Translation Quality Assessment: Evaluation and Estimation 22 / 33
38 Shared Task WMT12-13 with Radu Soricut & Christian Buck Sentence- and word-level estimation of PE effort Datasets and language pairs: Quality Year Languages 1-5 subjective scores WMT12 en-es Ranking all sentences best-worst WMT12/13 en-es HTER scores WMT13 en-es Post-editing time WMT13 en-es Word-level edits: change/keep WMT13 en-es Word-level edits: keep/delete/replace WMT13 en-es Ranking 5 MTs per source WMT13 en-es; de-en Evaluation metric: MAE = N i=1 H(s i) V (s i ) N Translation Quality Assessment: Evaluation and Estimation 22 / 33
39 Baseline system Features: number of tokens in the source and target sentences average source token length average number of occurrences of words in the target number of punctuation marks in source and target sentences LM probability of source and target sentences average number of translations per source word % of source 1-grams, 2-grams and 3-grams in frequency quartiles 1 and 4 % of seen source unigrams Translation Quality Assessment: Evaluation and Estimation 23 / 33
40 Baseline system Features: number of tokens in the source and target sentences average source token length average number of occurrences of words in the target number of punctuation marks in source and target sentences LM probability of source and target sentences average number of translations per source word % of source 1-grams, 2-grams and 3-grams in frequency quartiles 1 and 4 % of seen source unigrams SVM regression with RBF kernel with the parameters γ, ɛ and C optimised using a grid-search and 5-fold cross validation on the training set Translation Quality Assessment: Evaluation and Estimation 23 / 33
41 Results - scoring sub-task (WMT12) System ID MAE RMSE SDLLW M5PbestDeltaAvg UU best SDLLW SVM UU bltk Loria SVMlinear UEdin TCD M5P-resources-only* Baseline bb17 SVR Loria SVMrbf SJTU WLV-SHEF FS PRHLT-UPV WLV-SHEF BL DCU-SYMC unconstrained DFKI grcfs-mars DFKI cfs-plsreg UPC DCU-SYMC constrained UPC TCD M5P-all Translation Quality Assessment: Evaluation and Estimation 24 / 33
42 Results - scoring sub-task (WMT13) System ID MAE RMSE SHEF FS SHEF FS-AL CNGL SVRPLS LIMSI DCU-SYMC combine DCU-SYMC alltypes CMU nob CNGL SVR FBK-UEdin extra FBK-UEdin rand-svr LORIA inctrain Baseline bb17 SVR TCD-CNGL open LORIA inctraincont TCD-CNGL restricted CMU full UMAC Translation Quality Assessment: Evaluation and Estimation 25 / 33
43 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 26 / 33
44 Agreement between annotators Absolute value judgements: difficult to achieve consistency even in highly controlled settings 30% of initial dataset discarded Remaining annotations had to be scaled Translation Quality Assessment: Evaluation and Estimation 27 / 33
45 Agreement between annotators Absolute value judgements: difficult to achieve consistency even in highly controlled settings 30% of initial dataset discarded Remaining annotations had to be scaled More objective absolute scores Post-editing time, HTER, edits Also subject to huge variance (WPTP 2013, Wisniewski et. al, MT-Summit 2013) Multi-task learning to address this variance (Cohn and Specia, ACL 2013) Translation Quality Assessment: Evaluation and Estimation 27 / 33
46 Agreement between annotators Absolute value judgements: difficult to achieve consistency even in highly controlled settings 30% of initial dataset discarded Remaining annotations had to be scaled More objective absolute scores Post-editing time, HTER, edits Also subject to huge variance (WPTP 2013, Wisniewski et. al, MT-Summit 2013) Multi-task learning to address this variance (Cohn and Specia, ACL 2013) Relative scores Different task altogether WMT13: better results than reference-based metrics Translation Quality Assessment: Evaluation and Estimation 27 / 33
47 Annotation costs Active learning to select subset of instances to be annotated (Beck et al., ACL 2013) Translation Quality Assessment: Evaluation and Estimation 28 / 33
48 Curse of dimensionality Feature selection to identify relevant info for dataset (Shah et al., MT Summit 2013) 0.7' 0.65' 0.6' 0.55' 0.5' 0.45' 0.4' 0.35' ' ' 0.629' ' ' ' ' 0.449' ' 0.432' ' ' ' ' 0.411' ' ' ' ' ' ' ' ' ' 0.441' ' ' ' ' 0.506' ' ' ' ' 0.337' ' ' ' ' 0.521' 0.554' ' ' ' ' 0.3' WMT12' EAMT211'(en2es)' EAMT211'(fr2en)' EAMT2092s1' EAMT2092s2' EAMT2092s3' EAMT2092s4' GALE112s1' GALE112s2'' BL' AF' BL+PR' AF+PR' FS' Translation Quality Assessment: Evaluation and Estimation 29 / 33
49 Curse of dimensionality Feature selection to identify relevant info for dataset (Shah et al., MT Summit 2013) 0.7' 0.65' 0.6' 0.55' 0.5' 0.45' 0.4' 0.35' ' ' 0.629' ' ' ' ' 0.449' ' 0.432' ' ' ' ' 0.411' ' ' ' ' ' ' ' ' ' 0.441' ' ' ' ' 0.506' ' ' ' ' 0.337' ' ' ' ' 0.521' 0.554' ' ' ' ' 0.3' WMT12' EAMT211'(en2es)' EAMT211'(fr2en)' EAMT2092s1' EAMT2092s2' EAMT2092s3' EAMT2092s4' GALE112s1' GALE112s2'' BL' AF' BL+PR' AF+PR' FS' Common feature set identified, but nuanced subsets for specific datasets Translation Quality Assessment: Evaluation and Estimation 29 / 33
50 How to use estimated PE effort scores? Do users prefer detailed estimates (sub-sentence level) or an overall estimate for the complete sentence or not seeing bad sentences at all? Too much information vs hard-to-interpret scores Translation Quality Assessment: Evaluation and Estimation 30 / 33
51 How to use estimated PE effort scores? Do users prefer detailed estimates (sub-sentence level) or an overall estimate for the complete sentence or not seeing bad sentences at all? Too much information vs hard-to-interpret scores IBM s Goodness metric Translation Quality Assessment: Evaluation and Estimation 30 / 33
52 How to use estimated PE effort scores? Do users prefer detailed estimates (sub-sentence level) or an overall estimate for the complete sentence or not seeing bad sentences at all? Too much information vs hard-to-interpret scores IBM s Goodness metric MATECAT project investigating it Translation Quality Assessment: Evaluation and Estimation 30 / 33
53 Feature engineering Two families of features missing in current work: Can we benefit from contextual, document-wide information? Can we predict human translation quality? Translation Quality Assessment: Evaluation and Estimation 31 / 33
54 Feature engineering Two families of features missing in current work: Can we benefit from contextual, document-wide information? Can we predict human translation quality? Don t panic, you can help! Two (sub-) QuEst projects at MTM-2013 :-) Translation Quality Assessment: Evaluation and Estimation 31 / 33
55 Outline 1 Translation Quality 2 QTLaunchPad Project 3 Quality Estimation 4 State of the art in QE 5 Open issues 6 Conclusions Translation Quality Assessment: Evaluation and Estimation 32 / 33
56 Conclusions (Machine) Translation evaluation is still an open problem Translation Quality Assessment: Evaluation and Estimation 33 / 33
57 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Translation Quality Assessment: Evaluation and Estimation 33 / 33
58 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Quality estimation: learning of these different notions Translation Quality Assessment: Evaluation and Estimation 33 / 33
59 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Quality estimation: learning of these different notions Estimates have been used in real applications Ranking translations: filter out bad quality translations Selecting translations from multiple MT systems Translation Quality Assessment: Evaluation and Estimation 33 / 33
60 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Quality estimation: learning of these different notions Estimates have been used in real applications Ranking translations: filter out bad quality translations Selecting translations from multiple MT systems Commercial interest Translation Quality Assessment: Evaluation and Estimation 33 / 33
61 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Quality estimation: learning of these different notions Estimates have been used in real applications Ranking translations: filter out bad quality translations Selecting translations from multiple MT systems Commercial interest SDL LW: TrustScore Multilizer: MT-Qualifier Translation Quality Assessment: Evaluation and Estimation 33 / 33
62 Conclusions (Machine) Translation evaluation is still an open problem Different purposes/users, different needs, different notions of quality Quality estimation: learning of these different notions Estimates have been used in real applications Ranking translations: filter out bad quality translations Selecting translations from multiple MT systems Commercial interest SDL LW: TrustScore Multilizer: MT-Qualifier Interesting open issues: join the QuEst projects! Translation Quality Assessment: Evaluation and Estimation 33 / 33
Translation Quality Assessment: Evaluation and Estimation
Translation Quality Assessment: Evaluation and Estimation Lucia Specia University of Sheffield l.specia@sheffield.ac.uk MTM - Prague, 12 September 2016 Translation Quality Assessment: Evaluation and Estimation
Phrase-level Quality Estimation for Machine Translation
Phrase-level Quality Estimation for Machine Translation Varvara Logacheva, Lucia Specia University of Sheffield December 3, 2015 Varvara Logacheva, Lucia Specia Phrase-level Quality Estimation 1/21 Quality
Study of Translation Edit Rate with Targeted Human Annotation
Study of Translation Edit Rate with Targeted Human Annotation Matthew Snover (UMD) Bonnie Dorr (UMD) Richard Schwartz (BBN) Linnea Micciulla (BBN) John Makhoul (BBN) Outline Motivations Definition of Translation
Exploring Normalization Techniques for Human Judgments of Machine Translation Adequacy Collected Using Amazon Mechanical Turk
Exploring Normalization Techniques for Human Judgments of Machine Translation Adequacy Collected Using Amazon Mechanical Turk Michael Denkowski and Alon Lavie Language Technologies Institute School of
Choosing the Right Evaluation for Machine Translation: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks
Choosing the Right Evaluation for Machine Translation: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks Michael Denkowski and Alon Lavie Language Technologies Institute
Edinburgh s Neural Machine Translation Systems
Edinburgh s Neural Machine Translation Systems Barry Haddow University of Edinburgh October 27, 2016 Barry Haddow Edinburgh s NMT Systems 1 / 20 Collaborators Rico Sennrich Alexandra Birch Barry Haddow
Data Mining in Bioinformatics Day 4: Text Mining
Data Mining in Bioinformatics Day 4: Text Mining Karsten Borgwardt February 25 to March 10 Bioinformatics Group MPIs Tübingen Karsten Borgwardt: Data Mining in Bioinformatics, Page 1 What is text mining?
The Role of Face Parts in Gender Recognition
The Role of Face Parts in Gender Recognition Yasmina Andreu Ramón A. Mollineda Pattern Analysis and Learning Section Computer Vision Group University Jaume I of Castellón (Spain) Y. Andreu, R.A. Mollineda
Unsupervised Measurement of Translation Quality Using Multi-engine, Bi-directional Translation
Unsupervised Measurement of Translation Quality Using Multi-engine, Bi-directional Translation Menno van Zaanen and Simon Zwarts Division of Information and Communication Sciences Department of Computing
Further Meta-Evaluation of Machine Translation
Further Meta-Evaluation of Machine Translation Chris Callison-Burch Johns Hopkins University ccb cs jhu edu Cameron Fordyce camfordyce gmail com Philipp Koehn University of Edinburgh pkoehn inf ed ac uk
Sign Language MT. Sara Morrissey
Sign Language MT Sara Morrissey Introduction Overview Irish Sign Language Problems for SLMT SLMT Data MaTrEx for SLs Future work Introduction (1) Motivation SLs are poorly resourced and lack political,
Speech Processing / Speech Translation Case study: Transtac Details
Speech Processing 11-492/18-492 Speech Translation Case study: Transtac Details Phraselator: One Way Translation Commercial System VoxTec Rapid deployment Modules of 500ish utts Transtac: Two S2S System
Further Meta-Evaluation of Machine Translation
Further Meta-Evaluation of Machine Translation Chris Callison-Burch Johns Hopkins University ccb cs jhu edu Cameron Fordyce camfordyce gmail com Philipp Koehn University of Edinburgh pkoehn inf ed ac uk
Minimum Risk Training For Neural Machine Translation. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu
Minimum Risk Training For Neural Machine Translation Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu ACL 2016, Berlin, German, August 2016 Machine Translation MT: using computer
The Long Tail of Recommender Systems and How to Leverage It
The Long Tail of Recommender Systems and How to Leverage It Yoon-Joo Park Stern School of Business, New York University ypark@stern.nyu.edu Alexander Tuzhilin Stern School of Business, New York University
Measurement of Progress in Machine Translation
Measurement of Progress in Machine Translation Yvette Graham Timothy Baldwin Aaron Harwood Alistair Moffat Justin Zobel Department of Computing and Information Systems The University of Melbourne {ygraham,tbaldwin,aharwood,amoffat,jzobel}@unimelb.edu.au
Sentiment Analysis of Reviews: Should we analyze writer intentions or reader perceptions?
Sentiment Analysis of Reviews: Should we analyze writer intentions or reader perceptions? Isa Maks and Piek Vossen Vu University, Faculty of Arts De Boelelaan 1105, 1081 HV Amsterdam e.maks@vu.nl, p.vossen@vu.nl
Detection and Recognition of Sign Language Protocol using Motion Sensing Device
Detection and Recognition of Sign Language Protocol using Motion Sensing Device Rita Tse ritatse@ipm.edu.mo AoXuan Li P130851@ipm.edu.mo Zachary Chui MPI-QMUL Information Systems Research Centre zacharychui@gmail.com
Edinburgh Research Explorer
Edinburgh Research Explorer Further meta-evaluation of machine translation Citation for published version: Callison-Burch, C, Fordyce, C, Koehn, P, Monz, C & Schroeder, J 2008, Further meta-evaluation
Automated Estimation of mts Score in Hand Joint X-Ray Image Using Machine Learning
Automated Estimation of mts Score in Hand Joint X-Ray Image Using Machine Learning Shweta Khairnar, Sharvari Khairnar 1 Graduate student, Pace University, New York, United States 2 Student, Computer Engineering,
Language Technologies and Business in the Future
Language Technologies and Business in the Future Niko Papula, Multilizer KITES Symposium, Helsinki 31.10.2013 Finnish company providing leading translation & localization technologies since 1995 Each month
Image Captioning using Reinforcement Learning. Presentation by: Samarth Gupta
Image Captioning using Reinforcement Learning Presentation by: Samarth Gupta 1 Introduction Summary Supervised Models Image captioning as RL problem Actor Critic Architecture Policy Gradient architecture
Microblog Retrieval for Disaster Relief: How To Create Ground Truths? Ribhav Soni and Sukomal Pal
Microblog Retrieval for Disaster Relief: How To Create Ground Truths? Ribhav Soni and Sukomal Pal Outline 1. Overview of this work 2. Introduction 3. Background: FMT16 4. Experiments 5. Discussion Overview
Joint Emotion Analysis via Multi-task Gaussian Processes
Joint Emotion Analysis via Multi-task Gaussian Processes Daniel Beck Trevor Cohn Lucia Specia Department of Computer Science, University of Sheffield, United Kingdom {debeck1,l.specia}@sheffield.ac.uk
Modeling Annotator Rationales with Application to Pneumonia Classification
Modeling Annotator Rationales with Application to Pneumonia Classification Michael Tepper 1, Heather L. Evans 3, Fei Xia 1,2, Meliha Yetisgen-Yildiz 2,1 1 Department of Linguistics, 2 Biomedical and Health
Overview of the Patent Translation Task at the NTCIR-7 Workshop
Overview of the Patent Translation Task at the NTCIR-7 Workshop Atsushi Fujii, Masao Utiyama, Mikio Yamamoto, Takehito Utsuro University of Tsukuba National Institute of Information and Communications
J2.6 Imputation of missing data with nonlinear relationships
Sixth Conference on Artificial Intelligence Applications to Environmental Science 88th AMS Annual Meeting, New Orleans, LA 20-24 January 2008 J2.6 Imputation of missing with nonlinear relationships Michael
CLASSIFICATION OF BREAST CANCER INTO BENIGN AND MALIGNANT USING SUPPORT VECTOR MACHINES
CLASSIFICATION OF BREAST CANCER INTO BENIGN AND MALIGNANT USING SUPPORT VECTOR MACHINES K.S.NS. Gopala Krishna 1, B.L.S. Suraj 2, M. Trupthi 3 1,2 Student, 3 Assistant Professor, Department of Information
Prediction of Average and Perceived Polarity in Online Journalism
Prediction of Average and Perceived Polarity in Online Journalism Albert Chu, Kensen Shi, Catherine Wong Abstract We predicted the average and perceived journalistic objectivity in online news articles
Automatic Classification of Perceived Gender from Facial Images
Automatic Classification of Perceived Gender from Facial Images Joseph Lemley, Sami Abdul-Wahid, Dipayan Banik Advisor: Dr. Razvan Andonie SOURCE 2016 Outline 1 Introduction 2 Faces - Background 3 Faces
Exploiting Patent Information for the Evaluation of Machine Translation
Exploiting Patent Information for the Evaluation of Machine Translation Atsushi Fujii University of Tsukuba Masao Utiyama National Institute of Information and Communications Technology Mikio Yamamoto
Modeling Sentiment with Ridge Regression
Modeling Sentiment with Ridge Regression Luke Segars 2/20/2012 The goal of this project was to generate a linear sentiment model for classifying Amazon book reviews according to their star rank. More generally,
Evolutionary Programming
Evolutionary Programming Searching Problem Spaces William Power April 24, 2016 1 Evolutionary Programming Can we solve problems by mi:micing the evolutionary process? Evolutionary programming is a methodology
PR-SOCO Personality Recognition in SOurce COde 2016 Kolkata, 8-10 December
Personality Recognition in SOurce COde PAN@FIRE 2016 Kolkata, 8-10 December Francisco Rangel Fabio A. González & Felipe Restrepo-Calle Autoritas Consulting MindLab - Universidad Nacional Colombia Manuel
Prediction of heart disease using k-nearest neighbor and particle swarm optimization.
Biomedical Research 2017; 28 (9): 4154-4158 ISSN 0970-938X www.biomedres.info Prediction of heart disease using k-nearest neighbor and particle swarm optimization. Jabbar MA * Vardhaman College of Engineering,
A Web Tool for Building Parallel Corpora of Spoken and Sign Languages
A Web Tool for Building Parallel Corpora of Spoken and Sign Languages ALEX MALMANN BECKER FÁBIO NATANAEL KEPLER SARA CANDEIAS July 19,2016 Authors Software Engineer Master's degree by UFSCar CEO at Porthal
Memory-Augmented Active Deep Learning for Identifying Relations Between Distant Medical Concepts in Electroencephalography Reports
Memory-Augmented Active Deep Learning for Identifying Relations Between Distant Medical Concepts in Electroencephalography Reports Ramon Maldonado, BS, Travis Goodwin, PhD Sanda M. Harabagiu, PhD The University
EMOTION CLASSIFICATION: HOW DOES AN AUTOMATED SYSTEM COMPARE TO NAÏVE HUMAN CODERS?
EMOTION CLASSIFICATION: HOW DOES AN AUTOMATED SYSTEM COMPARE TO NAÏVE HUMAN CODERS? Sefik Emre Eskimez, Kenneth Imade, Na Yang, Melissa Sturge- Apple, Zhiyao Duan, Wendi Heinzelman University of Rochester,
Corpus Construction and Semantic Analysis of Indonesian Image Description
Corpus Construction and Semantic Analysis of Indonesian Image Description Khumaisa Nur aini 1,3, Johanes Effendi 1, Sakriani Sakti 1,2, Mirna Adriani 3, Sathosi Nakamura 1,2 1 Nara Institute of Science
Evaluating Classifiers for Disease Gene Discovery
Evaluating Classifiers for Disease Gene Discovery Kino Coursey Lon Turnbull khc0021@unt.edu lt0013@unt.edu Abstract Identification of genes involved in human hereditary disease is an important bioinfomatics
BayesOpt: Extensions and applications
BayesOpt: Extensions and applications Javier González Masterclass, 7-February, 2107 @Lancaster University Agenda of the day 9:00-11:00, Introduction to Bayesian Optimization: What is BayesOpt and why it
Utilizing Posterior Probability for Race-composite Age Estimation
Utilizing Posterior Probability for Race-composite Age Estimation Early Applications to MORPH-II Benjamin Yip NSF-REU in Statistical Data Mining and Machine Learning for Computer Vision and Pattern Recognition
Evaluating Quality in Creative Systems. Graeme Ritchie University of Aberdeen
Evaluating Quality in Creative Systems Graeme Ritchie University of Aberdeen Graeme Ritchie {2007} Some Empirical Criteria for Attributing Creativity to a Computer Program. Minds and Machines 17 {1}, pp.67-99.
Supplementary Creativity: Generating Diverse Questions using Variational Autoencoders
Supplementary Creativity: Generating Diverse Questions using Variational Autoencoders Unnat Jain UIUC uj2@illinois.edu Ziyu Zhang Northwestern University zzhang@u.northwestern.edu Alexander Schwing UIUC
Building Evaluation Scales for NLP using Item Response Theory
Building Evaluation Scales for NLP using Item Response Theory John Lalor CICS, UMass Amherst Joint work with Hao Wu (BC) and Hong Yu (UMMS) Motivation Evaluation metrics for NLP have been mostly unchanged
Automatic Medical Coding of Patient Records via Weighted Ridge Regression
Sixth International Conference on Machine Learning and Applications Automatic Medical Coding of Patient Records via Weighted Ridge Regression Jian-WuXu,ShipengYu,JinboBi,LucianVladLita,RaduStefanNiculescuandR.BharatRao
Generalizing Dependency Features for Opinion Mining
Generalizing Dependency Features for Mahesh Joshi 1 and Carolyn Rosé 1,2 1 Language Technologies Institute 2 Human-Computer Interaction Institute Carnegie Mellon University ACL-IJCNLP 2009 Short Papers
Handwriting - marker for Parkinson s Disease
Handwriting - marker for Parkinson s Disease P. Drotár et al. Signal Processing Lab Department of Telecommunications Brno University of Technology 3rd SPLab Workshop, 2013 P. Drotár et al. (Brno University
Affective Impact of Movies: Task Overview and Results
Affective Impact of Movies: Task Overview and Results Mats Sjöberg, Yoann Baveye, Hanli Wang, Vu Lam Quang, Bogdan Ionescu, Emmanuel Dellandréa, Markus Schedl, Claire-Hélène Demarty, Liming Chen MediaEval
D4.10 Demonstrator 4 EoT Application
Horizon 2020 PROGRAMME ICT-01-2014: Smart Cyber-Physical Systems This project has received funding from the European Union s Horizon 2020 research and innovation programme under Grant Agreement No 643924
The TUNA-REG Challenge 2009: Overview and Evaluation Results
The TUNA-REG Challenge 2009: Overview and Evaluation Results Albert Gatt Computing Science University of Aberdeen Aberdeen AB24 3UE, UK a.gatt@abdn.ac.uk Anja Belz Eric Kow Natural Language Technology
- - Xiaofen Xing, Bolun Cai, Yinhu Zhao, Shuzhen Li, Zhiwei He, Weiquan Fan South China University of Technology
- - - - -- Xiaofen Xing, Bolun Cai, Yinhu Zhao, Shuzhen Li, Zhiwei He, Weiquan Fan South China University of Technology 1 Outline Ø Introduction Ø Feature Extraction Ø Multi-modal Hierarchical Recall Framework
ML LAId bare. Cambridge Wireless SIG Meeting. Mary-Ann & Phil Claridge 23 November
ML LAId bare Cambridge Wireless SIG Meeting Mary-Ann & Phil Claridge 23 November 2017 www.mandrel.com @MandrelSystems info@mandrel.com 1 Welcome To Our Toolbox Our Opinionated Views! Data IDE Wrangling
Development of an Automated Language Interpreting Tool for Hispanic Mental Health Patients
Development of an Automated Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: Advancing Mental Health Equity in California March 30, 2017 Colaborators Steven Chan Alberto
Hierarchical Age Estimation from Unconstrained Facial Images
Hierarchical Age Estimation from Unconstrained Facial Images STIC-AmSud Jhony Kaesemodel Pontes Department of Electrical Engineering Federal University of Paraná - Supervisor: Alessandro L. Koerich (/PUCPR
May All Your Wishes Come True: A Study of Wishes and How to Recognize Them
May All Your Wishes Come True: A Study of Wishes and How to Recognize Them Andrew B. Goldberg, Nathanael Fillmore, David Andrzejewski, Zhiting Xu, Bryan Gibson & Xiaojin Zhu Computer Sciences Department
IMPORTANCE OF AND CRITERIA FOR EVALUATING EXTERNAL VALIDITY. Russell E. Glasgow, PhD Lawrence W. Green, DrPH
IMPORTANCE OF AND CRITERIA FOR EVALUATING EXTERNAL VALIDITY Russell E. Glasgow, PhD Lawrence W. Green, DrPH 1 OVERVIEW Rationale for and Importance of External Validity (EV) for Translation Proposed EV
Microphone Input LED Display T-shirt
Microphone Input LED Display T-shirt Team 50 John Ryan Hamilton and Anthony Dust ECE 445 Project Proposal Spring 2017 TA: Yuchen He 1 Introduction 1.2 Objective According to the World Health Organization,
Gene Selection for Tumor Classification Using Microarray Gene Expression Data
Gene Selection for Tumor Classification Using Microarray Gene Expression Data K. Yendrapalli, R. Basnet, S. Mukkamala, A. H. Sung Department of Computer Science New Mexico Institute of Mining and Technology
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability Jonathan H. Clark Chris Dyer Alon Lavie Noah A. Smith Language Technologies Institute Carnegie Mellon
Chapter 12 Conclusions and Outlook
Chapter 12 Conclusions and Outlook In this book research in clinical text mining from the early days in 1970 up to now (2017) has been compiled. This book provided information on paper based patient record
Search e Fall /18/15
Sample Efficient Policy Click to edit Master title style Search Click to edit Emma Master Brunskill subtitle style 15-889e Fall 2015 11 Sample Efficient RL Objectives Probably Approximately Correct Minimizing
Annotation and Retrieval System Using Confabulation Model for ImageCLEF2011 Photo Annotation
Annotation and Retrieval System Using Confabulation Model for ImageCLEF2011 Photo Annotation Ryo Izawa, Naoki Motohashi, and Tomohiro Takagi Department of Computer Science Meiji University 1-1-1 Higashimita,
Comparative Analysis of Machine Learning Algorithms for Chronic Kidney Disease Detection using Weka
I J C T A, 10(8), 2017, pp. 59-67 International Science Press ISSN: 0974-5572 Comparative Analysis of Machine Learning Algorithms for Chronic Kidney Disease Detection using Weka Milandeep Arora* and Ajay
Semi-Automatic Construction of Thyroid Cancer Intervention Corpus from Biomedical Abstracts
jsci2016 Semi-Automatic Construction of Thyroid Cancer Intervention Corpus from Biomedical Wutthipong Kongburan, Praisan Padungweang, Worarat Krathu, Jonathan H. Chan School of Information Technology King
Accessible Computing Research for Users who are Deaf and Hard of Hearing (DHH)
Accessible Computing Research for Users who are Deaf and Hard of Hearing (DHH) Matt Huenerfauth Raja Kushalnagar Rochester Institute of Technology DHH Auditory Issues Links Accents/Intonation Listening
What Predicts Media Coverage of Health Science Articles?
What Predicts Media Coverage of Health Science Articles? Byron C Wallace, Michael J Paul & Noémie Elhadad University of Texas at Austin; byron.wallace@utexas.edu! Johns Hopkins University! Columbia University!
How preferred are preferred terms?
How preferred are preferred terms? Gintare Grigonyte 1, Simon Clematide 2, Fabio Rinaldi 2 1 Computational Linguistics Group, Department of Linguistics, Stockholm University Universitetsvagen 10 C SE-106
Intelligent Machines That Act Rationally. Hang Li Toutiao AI Lab
Intelligent Machines That Act Rationally Hang Li Toutiao AI Lab Four Definitions of Artificial Intelligence Building intelligent machines (i.e., intelligent computers) Thinking humanly Acting humanly Thinking
Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.1 7/27/2011
Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.1 7/27/2011 I. Purpose Drawing from the profile development of the QIBA-fMRI Technical Committee,
Experiments with CST-based Multi-document Summarization
Experiments with CST-based Multi-document Summarization Maria Lucia R. Castro Jorge mluciacj@icmc.usp.br Thiago A.S. Pardo taspardo@icmc.usp.br NLP Group - University of São Paulo Multi-document Summarization
Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education.
The Reliability of PLATO Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO M. Ken Cor Stanford University School of Education April,
Predicting clinical outcomes in neuroblastoma with genomic data integration
Predicting clinical outcomes in neuroblastoma with genomic data integration Ilyes Baali, 1 Alp Emre Acar 1, Tunde Aderinwale 2, Saber HafezQorani 3, Hilal Kazan 4 1 Department of Electric-Electronics Engineering,
THE data used in this project is provided. SEIZURE forecasting systems hold promise. Seizure Prediction from Intracranial EEG Recordings
1 Seizure Prediction from Intracranial EEG Recordings Alex Fu, Spencer Gibbs, and Yuqi Liu 1 INTRODUCTION SEIZURE forecasting systems hold promise for improving the quality of life for patients with epilepsy.
Smaller, faster, deeper: University of Edinburgh MT submittion to WMT 2017
Smaller, faster, deeper: University of Edinburgh MT submittion to WMT 2017 Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, Philip
Illinois Supreme Court. Language Access Policy
Illinois Supreme Court Language Access Policy Effective October 1, 2014 ILLINOIS SUPREME COURT LANGUAGE ACCESS POLICY I. PREAMBLE The Illinois Supreme Court recognizes that equal access to the courts is
Knowledge Discovery and Data Mining I
Ludwig-Maximilians-Universität München Lehrstuhl für Datenbanksysteme und Data Mining Prof. Dr. Thomas Seidl Knowledge Discovery and Data Mining I Winter Semester 2018/19 Introduction What is an outlier?
Review of SPRACH/Thisl meetings Cambridge UK, 1998sep03/04
Review of SPRACH/Thisl meetings Cambridge UK, 1998sep03/04 Dan Ellis International Computer Science Institute, Berkeley CA Outline 1 2 3 4 5 SPRACH overview The SPRACH Broadcast
Colon cancer subtypes from gene expression data
Colon cancer subtypes from gene expression data Nathan Cunningham Giuseppe Di Benedetto Sherman Ip Leon Law Module 6: Applied Statistics 26th February 2016 Aim Replicate findings of Felipe De Sousa et
Continuous Measurement Scales in Human Evaluation of Machine Translation
Continuous Measurement Scales in Human Evaluation of Machine Translation Yvette Graham Timothy Baldwin Alistair Moffat Justin Zobel Department of Computing and Information Systems, The University of Melbourne
Discovering Meaningful Cut-points to Predict High HbA1c Variation
Proceedings of the 7th INFORMS Workshop on Data Mining and Health Informatics (DM-HI 202) H. Yang, D. Zeng, O. E. Kundakcioglu, eds. Discovering Meaningful Cut-points to Predict High HbAc Variation Si-Chi
Multi-modal Patient Cohort Identification from EEG Report and Signal Data
Multi-modal Patient Cohort Identification from EEG Report and Signal Data Travis R. Goodwin and Sanda M. Harabagiu The University of Texas at Dallas Human Language Technology Research Institute http://www.hlt.utdallas.edu
Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Ryan Adams, Hugo LaRochelle NIPS 2012
Practical Bayesian Optimization of Machine Learning Algorithms Jasper Snoek, Ryan Adams, Hugo LaRochelle NIPS 2012 ... (Gaussian Processes) are inadequate for doing speech and vision. I still think they're
Extracting geographic locations from the literature for virus phylogeography using supervised and distant supervision methods
Extracting geographic locations from the literature for virus phylogeography using supervised and distant supervision methods D. Weissenbacher 1, A. Sarker 2, T. Tahsin 1, G. Gonzalez 2 and M. Scotch 1
Intelligent Machines That Act Rationally. Hang Li Bytedance AI Lab
Intelligent Machines That Act Rationally Hang Li Bytedance AI Lab Four Definitions of Artificial Intelligence Building intelligent machines (i.e., intelligent computers) Thinking humanly Acting humanly
Intro to HCI / Why is Design Hard?
Intro to HCI / Why is Design Hard? September 11, 2017 Fall 2017 COMP 3020 1 Fall 2017 COMP 3020 2 Announcements Assignment 1 is posted Due Sept 22 by 5:00pm on UMLearn Individual assignment Buying Pop
CHAPTER 4 THE QUESTIONNAIRE DESIGN /SOLUTION DESIGN. This chapter contains explanations that become a basic knowledge to create a good
CHAPTER 4 THE QUESTIONNAIRE DESIGN /SOLUTION DESIGN This chapter contains explanations that become a basic knowledge to create a good questionnaire which is able to meet its objective. Just like the thesis
Incorporating qualitative research into guideline development: the way forward
11 Dec 2015, Baveno Incorporating qualitative research into guideline development: the way forward Ӧzge Tunçalp, MD PhD Department of Reproductive Health and Research 1 Outline WHO guideline processes
Background Information
Background Information Erlangen, November 26, 2017 RSNA 2017 in Chicago: South Building, Hall A, Booth 1937 Artificial intelligence: Transforming data into knowledge for better care Inspired by neural
User Guide V: 3.0, August 2017
User Guide V: 3.0, August 2017 a product of FAQ 3 General Information 1.1 System Overview 5 1.2 User Permissions 6 1.3 Points of Contact 7 1.4 Acronyms and Definitions 8 System Summary 2.1 System Configuration
Gender Based Emotion Recognition using Speech Signals: A Review
50 Gender Based Emotion Recognition using Speech Signals: A Review Parvinder Kaur 1, Mandeep Kaur 2 1 Department of Electronics and Communication Engineering, Punjabi University, Patiala, India 2 Department
Toward the Evaluation of Machine Translation Using Patent Information
Toward the Evaluation of Machine Translation Using Patent Information Atsushi Fujii Graduate School of Library, Information and Media Studies University of Tsukuba Mikio Yamamoto Graduate School of Systems
1. INTRODUCTION. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 1
1. INTRODUCTION Sign language interpretation is one of the HCI applications where hand gesture plays important role for communication. This chapter discusses sign language interpretation system with present
Bayesian Kernel Methods for Natural Language Processing
Bayesian Kernel Methods for Natural Language Processing Daniel Beck Department of Computer Science University of Sheffield Sheffield, United Kingdom debeck1@sheffield.ac.uk Abstract Kernel methods are
Vital Responder: Real-time Health Monitoring of First- Responders
Vital Responder: Real-time Health Monitoring of First- Responders Ye Can 1,2 Advisors: Miguel Tavares Coimbra 2, Vijayakumar Bhagavatula 1 1 Department of Electrical & Computer Engineering, Carnegie Mellon
Diagnosis of Breast Cancer Using Ensemble of Data Mining Classification Methods
International Journal of Bioinformatics and Biomedical Engineering Vol. 1, No. 3, 2015, pp. 318-322 http://www.aiscience.org/journal/ijbbe ISSN: 2381-7399 (Print); ISSN: 2381-7402 (Online) Diagnosis of
High-fidelity Imaging Response Detection in Multiple Sclerosis
High-fidelity Imaging Response Detection in Multiple Sclerosis Dr Baris Kanber, Translational Imaging Group, Centre for Medical Image Computing, Department of Medical Physics and Biomedical Engineering,
State Education Agency Accessibility and Accommodations Policies:
State Education Agency Accessibility and Policies: 2017-2018 Within the ACCESS for ELLs 2.0 Accessibility and Supplement, there are a number of places where the text refers to the development of specific
Discovery and Clinical Decision Support for Personalized Healthcare
1 Discovery and Clinical Decision Support for Personalized Healthcare Jinsung Yoon, Camelia Davtyan, MD, FACP, and Mihaela van der Schaar, Fellow, IEEE Abstract With the advent of electronic health records,
EECS 433 Statistical Pattern Recognition
EECS 433 Statistical Pattern Recognition Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208 http://www.eecs.northwestern.edu/~yingwu 1 / 19 Outline What is Pattern