Open Architectures and Community Building. Professor Joakim Gustafson. Head of Department Department of Speech, Music and Hearing KTH, Sweden

Size: px

Start display at page:

Download "Open Architectures and Community Building. Professor Joakim Gustafson. Head of Department Department of Speech, Music and Hearing KTH, Sweden"

Matthew Weaver
5 years ago
Views:

1 Open Architectures and Community Building Professor Joakim Gustafson Head of Department Department of Speech, Music and Hearing KTH, Sweden

2 The KTH speech group s areas of interest Human speech perception/production Situated face-to-face interaction Tools and architectures Data collections and corpora

3 Multimodal dialogue systems

Collecting human-(human) speech corpora Speechdat, Specom & PF-Star read

Spontal dyadic dialogue speech corpus -120 spontaneous human-human dialogues

KTH-IDAP multiparty dialogue speech corpus -5 one-hour multisensory

4 Collecting human-(human) speech corpora Speechdat, Specom & PF-Star read speach corpora - Scripted speech material for synthesis and recognition Spontal dyadic dialogue speech corpus -120 spontaneous human-human dialogues with video and motion capture for speech and gestures. KTH-IDAP multiparty dialogue speech corpus -5 one-hour multisensory recordings of four-party interactions with automatic generation of voice activity, eye gaze and head nod annotations.

5 SpeDial: Spoken Dialogue Analytics Devise machine-aided algorithms for spoken dialogue system enhancement and customization for call-center applications Partners: NuEcho,VoiceWeb, KTH, INESC-ID,TSI-TUC

6 Corpora used in the SpeDial project Dataset name Language Owner Free Movie ticketing Bank application CORAL Switchboard Cambridge Restaurant Let s Go SweCC Lost baggage Greek Greek Portuguese English English English Swedish English Internal commercial Internal commercial Internal academic External academic External academic External academic External commercial External commercial No No Yes Yes Yes Yes No No 6

7 SPDXml: Common dialogue log format XML file format defined by an XSD document Encodes complete dialogs : - Dialogue attributes Eg. caller ID, goals, audio, estimated success, gender, age, -All turn exchanges (system & user) State name Prompts Recognition results (ASR hypotheses, semantic hypotheses, etc.) Transcription All sorts of annotations affective speech disfluencies hot spots

8 -an open source dialogue framework Speech recognition Semantic parsing Dialogue manager Event driven Statechart-based XML for dialogue flow design Speech synthesis Face tracking Head pose/gaze Expressions Identity System Touch screen Display Facial animation Motors Developed by Gabriel Skantze

9 Speech APIs action.listen :endsil :grammar action.listen.stop Speech recognition Microsoft Nuance 9 Sphinx (open source) Nuance Dragon Google sense.speech.start sense.speech.rec.partial sense.speech.end sense.speech.rec.final :text :sem action.speech :text action.speech.stop Speech synthesis Windows (SAPI) Cereproc Mary TTS (open source) montor.speech.start montor.speech.end

10 Avatar APIs action.lipsync :phones monitor.lipsync action.gaze :location action.gesture :name LipSpeaker Furhat monitor.gaze monitor.gesture EMBR IURO

11 Swedish CLARIN Infrastructure center -The Swedish National Data Service Service-providing center -Språkbanken(the Language Bank) Knowledge Centers - Språkrådet(the Language Council) -Digisam -KTH -Lund University -Stockholm University - Uppsala University A common project to identify speech corpora at institutes and gov. agencies

Development of a national spoken language bank for speech-based services Would it be possible (and worthwhile) set up a Swedish spoken language bank that is freely available for

12 Development of a national spoken language bank for speech-based services Would it be possible (and worthwhile) set up a Swedish spoken language bank that is freely available for commercial use in the area of e-participation/e-inclusion? A Swedish government commision tothe Swedish Post and Telecom Authority(PTS) KTH is currently doing the investigation on behalf of PTS

13 Spoken language resource descriptions Quantity, quality, type of material Assessment of its usability for speech technology research and commercial application development Ownership form, contracts, distribution terms, licenses Other legal relations, integrity agreements, ethical considerations Description of the metadata, annotation, file format Physical accessibility

that will be launched on a mediawiki server and build voices for Swedish, English and a third language

14 WikiSpeech a WikiMedia-KTH-STTS project application to PTS Step 1: Investigate how to develop an cloud-based speech synthesizer and web-tools for recording and building voices Step 2: Develop platform that will be launched on a mediawiki server and build voices for Swedish, English and a third language Legal aspects: - All software open source: -All data (sound recordings, text, dictionaries, grammars) freely available:

Accessible Computing Research for Users who are Deaf and Hard of Hearing (DHH)

Accessible Computing Research for Users who are Deaf and Hard of Hearing (DHH) Matt Huenerfauth Raja Kushalnagar Rochester Institute of Technology DHH Auditory Issues Links Accents/Intonation Listening