ATLAS. Automatic Translation Into Sign Languages

Transcription

1 ATLAS Automatic Translation Into Sign Languages Gabriele TIOTTO Politecnico di Torino (Italy)

2 Inclusion E-INCLUSION is an important step for both improving people access to the information society and overcoming social exclusions.

3 Deaf People The integration difficulties of persons who were born deaf or acquired deafness in the first years of life are higher because they could not acquire the knowledge of the spoken language. More than half of deaf people encounter difficulties in reading the written form of any spoken language and, as a result, suffer from sub-education. Furthermore, almost all deaf people are unable to hear or read lips well enough to understand a spoken language as it is spoken. Hence, deaf people have difficulty conversing with non -signers, watching movies, television or reading books.

4 Sign Languages

5 Sign Languages A Sign Language (SL) is a visual language based on body gestures instead of sound to convey meaning. It simultaneously combines shapes, orientations and movements of the hands, as well as non-manual components, like facial expressions. SLs are not pantomime, nor are they a visual rendition of the related Verbal Language.

6 Sign Languages (2) Sign Languages are real languages like any other human languages and not just pictures in the air. The European Parliament, on June 17th, 1988 and November 18th, 1998, and the UNESCO in Salamanca Resolution, art. 21, on May 15th, 2001, stated the official status of Sign Languages of Deaf people as official languages.

7 Sign Languages (3) Wherever communities of deaf people exist, SLs develop. Sign Languages allow deaf children to acquire a full cognitive development within their community composed of hearing people and deaf persons. Such cognitive development represents the basic prerequisite to a full access to the education, the culture, and the inclusion in working and social environments.

8 Sign Language is not Universal Lingua dei segni italiana (LIS) American Sign Language (ASL) British Sign Language (BSL) Lingua dei segni francese (LSF) Lingua dei segni nicaraguense (ISN) Lingua dei segni neozelandese (NZSL) Gestuno (un lingua artificiale parzialmente condivisa e codificato) Signuno (un lingua dei segni basato sull'esperanto) Finger spelling

9 Deaf people in Italy In Italy 1 person out of 1,000 is affected by serious deafness In Italy there are today about 50,000 persons who were born deaf or got deafness in the first years of life

10 Private vs Public Investments The number of involved people are remarkable, but unfortunately not enough to motivate serious private investments.

11 ATLAS : what is it a 3-year research project, started January 15th, 2009 co-funded by Regione Piemonte, within the Converging Technologies - CIPE 2007 framework (Research Sector: Cognitive Science and ICT)

12 ATLAS : Goals The ATLAS project aims at translating written Italian language to Italian Sign Language and at visualizing it on different media via virtual characters.

13

14

15 ATLAS : The partners Politecnico di Torino - Dip di Automatica e Informatica Università di Torino - Dip di Informatica Università di Torino - Dip di Psicologia RAI Radiotelevisione Italiana S.p.A. BEPS Engineering Microsoft Innovation Center - ISMB Virtual Reality and Multimedia Park Lumiq Studios s.r.l. Alto Sistemi s.r.l. CSP - Innovazione nelle ICT Fondazione Bruno Kessler Trento Cooperativa scs Global Comunication University of Illinois at Chicago.

16 ATLAS : Why? LIS <> written Italian

17 ATLAS : Why? LIS <> written Italian Human Interpreter <> Virtual Interpreter

18 Plus of a Virtual Interpreter Cheaper Technologically Advanced New Services Ergonomic

19 Plus of a Virtual Interpreter Can be easily enabled /disabled

20 Plus of a Virtual Interpreter Can be freely customized according to user s preferences:

21 Plus of a Virtual Interpreter Can be customized for deaf vision-impaired people. For example one can set a high contrast visualization or particular chromatic settings to compensate her/his specific sight defects :

22 Questionnaire We set up a questionnaire on to gather preferences and suggestions

23 ATLAS : Why? LIS <> written Italian Human Interpreter <> Virtual Interpreter Just TV <> Several Media

24 An example of Application for Mobile At the railway station or at the airport provide a service it visualizes on cell phones the same messages announced by the speaker.

25 An example of Application for the web

26 An example of Application for the movies

27 2 7 State-of-the-Art English-ASL (USA): no grammar, 1 to 1 translation SignSpeak (EU): sign to sign translation Blue Sign (IT): Italian LIS, poor quality

28 ATLAS : Why a research project? A plenty of technological and scientific challeges to be faced

29 ATLAS : Scientific challenges Formal representation of the LIS Automatic translation from Italian to LIS Creation of linguistic corpora Definition of a virtual character with proper expressions Transmission/visualization on several media,,,

30 ATLAS Workflow TEXT VIDEO VIRTUAL ACTOR COMMANDS SOURCES Teletext Subtitles Audio Audio/Video TEXT AEWLIS Sentences Virtual Actor Commands

31 ATLAS Translation Pipeline Italian Text NL to AEWLIS Translator AEWLIS Text AEWLIS to AL Translator AL Text Input texts written in Italian natural language are first translated into an intermediate representation, called ATLAS Extended Written Italian Sign Language (AEWLIS)

32 ATLAS Translation Pipeline Italian Text NL to AEWLIS Translator AEWLIS Text AEWLIS to AL Translator AL Text Sentences expressed in AEWLIS are then translated into character's gestures Animation Language (AL) which describes the way the basic movements are produced and linked.

33 ATLAS Translation Pipeline AL Text 3D Rendering Engine Screen Motion capture Keyframing Procedural Animation Techniques The resulting AL sequences are rendered through a 3D rendering engine. An automatic animation blending system ensures that the generated LIS animations include smooth transitions between signs.

34 SL Translation Issues Translating from a spoken language to a SL is a complex undertaking: SL signers communicate through concepts, not words Many signs are iconic SLs incorporate gestures, facial expressions, head movements, body language and even the space around the speaker SL signers use the space around them for several grammatical, discourse, and descriptive purposes: classifier predicates. Important concepts: signing space model, timing model, fingerspelling

35 Italian Sign Language (LIS) Having good knowledge of the peculiarities of the LIS is important when building LIS animations: Existence of several LIS dialects Linguistics: Agreement between nouns, adjectives and verbs, which it is not based on gender but on place Facial expressions replace voice intonations Phonological distinctions between nouns and verbs; Plurals and concordance with verbs, gender, Fingerspelling: spelling out the word using gestures that correspond to the letters, usually used when no standard sign exists for a desired word

36 Representing SLs Translation prototypically involves getting a text in one language and providing it in another. SLs have no standard written form Problem: State-of-the-art MT systems use the word-for -word approach, producing translations that lack of syntax Solution: abstract from the sign synthesis problem by using a writing system Existing writing systems: HamNoSys Stokoe Notation Movement-Hold model Radutzky s Notation AEWLIS

37 ATLAS : User profiling ATLAS users are classified according to their role, distinguishing between: ATLAS partners ATLAS Power Users, i.e., specialized users charged of creating the corpora needed to support and improve the statistical translation phase. ATLAS Target Users, i.e., deaf people that will benefit from the availability of the ATLAS virtual interpreters on several communication media, including digital TV, internet, DVDs, smart-phones, All Users, which can access, via the ATLAS portal, all the products and by-products developed within the project.

38 ATLAS : Possible fall-outs Opportunities for New services E-inclussion New professions All the developed tools made available via open source to the scientific community

39 ATLAS : By-products LIS Dictionary for Smart Phones Annotation tool for Corpora creation Applications for the on-demand translation Service Portal Porting of OGRE on Windows CE, available and open source Assisted editor for manual translation

40 ATLAS : The first results

41 ATLAS : Donna with facial expressions

42