Speech Processing / Speech Translation Case study: Transtac Details

Size: px

Start display at page:

Download "Speech Processing / Speech Translation Case study: Transtac Details"

Lewis Ferguson
5 years ago
Views:

1 Speech Processing / Speech Translation Case study: Transtac Details

2 Phraselator: One Way Translation Commercial System VoxTec Rapid deployment Modules of 500ish utts

3 Transtac: Two S2S System DARPA developed for Check points, medical and civil defense Requirements Two way Eyes-free (no screen) Portable Usable by real users

4 Transtac System Close-talking Microphone Optional speech control Push-to-Talk Buttons Small powerful Speakers Laptop secured in Backpack

5 Transtac System Details Two way system 2 ASR systems: English and Iraqi 2 way statistical translation 2 synthesizers Push-to-talk system (Users don t like translate everything mode ) Echo back ASR result And then translation

6 Iraqi Language Iraqi Arabic is a dialect Most Iraqi s write Modern Standard Arabic Most Iraqi s do not write their own dialect No standardized spelling Transtac project invented one But Iraqi s may not be used to it Arabic (MSA and dialects) Do not write short vowels in words

7 Data for Training Collected human mediated dialogs Human acts as a machine Passed a microphone back an forward Try to get people not to talk at same time Large number of collections (over 4 years) 650 thousand sentences pairs Many different speakers Hand transcribed by experts (in Iraqi spelling) Hand translate (Source sentences and Interpreter s)

Iraqi ASR Acoustic model from Iraqi data Based on MSA phoneset Needs to be small fast models Discriminative Training Speaker specific adaptation Lexicon Based

8 Iraqi ASR Acoustic model from Iraqi data Based on MSA phoneset Needs to be small fast models Discriminative Training Speaker specific adaptation Lexicon Based on LDC provided lexicon Multiple pronunciations/typos still a problem Statistically trained LTS rules Language Model Trained on Iraqi input (and translated output)

English ASR Acoustic model Originally using other models Then trained from collected data (Mostly military personnel) Lexicon Existing lexicon but needed to

9 English ASR Acoustic model Originally using other models Then trained from collected data (Mostly military personnel) Lexicon Existing lexicon but needed to add Military speak: MRAP, IED Language model Trained from data provided Trained from similar data found on the web Training from hand created typical examples

10 TTS Standard English TTS Appropriate command voice Unit selection Added lots of military vocabulary Iraqi TTS Recorded from Iraqi radio announcer Based on example sentences in the domain LDC lexicon and LTS rules (same as ASR) Hand tuned

11 S2S Interface Issues How do you teach people to use the system Transtac say instructions Not really sufficient How can you tell it translated correctly Give (speech) feedback. Backtranslation ASR echo back

12 S2S Interface Issues How do you translate names A correct translation/transliteration is hard to understand Mark names in translations My name is Abdullah He lives on al-aqar street

13 S2S Evaluation (Transtac) Offline tests ASR->Text and Text->Text Compare to translation references WER and BLEU score Online tests Concept transfer (through defined scenarios) Speed (number of concepts per minute) (English speech masking) Utility tests Does it really work

14 Transtac Participants Developer groups IBM SRI BBN CMU USC Evaluations Twice a year in Iraqi (somewhere in DC) One surprise language Farsi, Bahasa Malay, Dari, Pashto Other evaluations with military groups

15 Does it work?? Yes, mostly 27 concepts out of 30-ish turns Systems are mostly similar But some better than others Other techniques Belt/holster based PC with handheld speaker Small PC in pouch Chest mounted array microphone

16 S2S ASR Advanced issues Tight coupling ASR should output N-best Translated all (lattice) Choose best translation (MT as a LM for ASR) Remove disfluencies/hestitations Add more relevant data Automatically convert past tense/third person data to present tense/first+second person

17 S2S TTS Advance Issues MT output isn t grammatical TTS doesn t care and just says it TTS should try to say MT output with more breaks. TTS (unit selection) As a LM on MT output Choose the best translation on what is said best

18 S2S MT Advanced issues Train on ASR output Do ASR on training data Build SMT model ASR-TEXT to TEXT Session adaptation Improve coverage from daily usage

19 S2S In-line Translation CMU-INESC (Portugal) project Translation of TED videos Align audio to give dubbing not voiceover Align: timing, breaks, focus across language

Toward Univeral Network-based Speech Translation

Toward Univeral Network-based Speech Translation Chai Wutiwiwatchai Speech and Audio Technology Laboratory National Electronics and Computer Technology Center 1 Outline Technology Review U-STAR Consortium