ATLAS Automatic Translation Into Sign Languages

Transcription

1 ATLAS Automatic Translation Into Sign Languages Leonardo LESMO e Alessandro MAZZEI Università di Torino Presentazione preparata in collaborazione con Paolo Prinetto, Politecnico di Torino

2 Deaf People (e-inclusion) The integration difficulties of persons who were born deaf or acquired deafness in the first years of life are higher because they could not acquire the knowledge of the spoken language. More than half of deaf people encounter difficulties in reading the written form of any spoken language and, as a result, suffer from sub-education. Furthermore, almost all deaf people are unable to hear or read lips well enough to understand a spoken language as it is spoken. Hence, deaf people have difficulty conversing with nonsigners, watching movies, television or reading books.

3 Sign Languages A Sign Language (SL) is a visual language based on body gestures instead of sound to convey meaning. It simultaneously combines shapes, orientations and movements of the hands, as well as non-manual components, like facial expressions. SLs are not pantomime (pictures in the air), nor are they a visual rendition of the related Verbal Language. The European Parliament, on June 17th, 1988 and November 18th, 1998, and the UNESCO in Salamanca Resolution, art. 21, on May 15th, 2001, stated the official status of Sign Languages of Deaf people as official languages.

4 Sign Language is not Universal Lingua dei segni italiana (LIS) American Sign Language (ASL) British Sign Language (BSL) French SL (LSF) Nicaragua SL (ISN) New Zealand SL (NZSL) Gestuno (artificial SL) Signuno (a SL based on Esperanto) Finger spelling

5 ATLAS : what is it In Italy 1 person out of 1,000 is affected by serious deafness; so, there are today about 50,000 persons who were born deaf or got deafness in the first years of life ATLAS is a 3-year research project, started January 15th, 2009 It was co-funded by Regione Piemonte, within the Converging Technologies - CIPE 2007 framework (Research Sector: Cognitive Science and ICT) Its goal was the translation of written Italian language into Italian Sign Language and its visualization on different media via virtual characters.

6

7

8 ATLAS : The partners Politecnico di Torino - Dip di Automatica e Informatica Università di Torino - Dip di Informatica Università di Torino - Dip di Psicologia RAI Radiotelevisione Italiana S.p.A. BEPS Engineering Microsoft Innovation Center - ISMB Virtual Reality and Multimedia Park Lumiq Studios s.r.l. Alto Sistemi s.r.l. CSP - Innovazione nelle ICT Fondazione Bruno Kessler Trento Cooperativa scs Global Comunication University of Illinois at Chicago.

9 Plus of a Virtual Interpreter Cheaper Technologically Advanced New Services Ergonomic

10 Plus of a Virtual Interpreter Can be freely customized according to user s preferences:

11 An example of Application for Mobile At the railway station or at the airport provide a service yo visualize on cell phones the same messages announced by the speaker.

12 State-of-the-Art English-ASL (USA): no grammar, 1 to 1 translation SignSpeak (EU): sign to sign translation Blue Sign (IT): Italian LIS, poor quality 1

13 ATLAS : Scientific challenges Formal representation of the LIS Automatic translation from Italian to LIS Creation of linguistic corpora Definition of a virtual character with proper expressions Transmission/visualization on several media,,,

14 ATLAS Workflow TEXT VIDEO VIRTUAL ACTOR COMMANDS SOURCES Teletext Subtitles Audio Audio/Video TEXT AEWLIS Sentences Virtual Actor Commands Transformations Transformations Transformations

15 ATLAS Translation Pipeline Italian Text NL to AEWLIS Translator AEWLIS Text AEWLIS to AL Translator AL Text Input texts written in Italian natural language are first translated into an intermediate representation, called ATLAS Extended Written Italian Sign Language (AEWLIS)

16 ATLAS Translation Pipeline Italian Text NL to AEWLIS Translator AEWLIS Text AEWLIS to AL Translator AL Text Sentences expressed in AEWLIS are then translated into character's gestures Animation Language (AL) which describes the way the basic movements are produced and linked.

17 ATLAS Translation Pipeline AL Text 3D Rendering Engine Screen Motion capture Keyframing Procedural Animation Techniques The resulting AL sequences are rendered through a 3D rendering engine. An automatic animation blending system ensures that the generated LIS animations include smooth transitions between signs.

18 SL Translation Issues Translating from a spoken language to a SL is a complex undertaking: SL signers communicate through concepts, not words Many signs are iconic SLs incorporate gestures, facial expressions, head movements, body language and even the space around the speaker SL signers use the space around them for several grammatical, discourse, and descriptive purposes: classifier predicates. Important concepts: signing space model, timing model, fingerspelling

19 Italian Sign Language (LIS) Having good knowledge of the main aspects and peculiarities of the LIS is important when building LIS animations: Existence of several LIS dialects Linguistics: Agreement between nouns, adjectives and verbs, which it is not based on gender but on place Facial expressions replace voice intonations Plurals and concordance with verbs Fingerspelling: spelling out the word using gestures that correspond to the letters, usually used when no standard sign exists for a desired word

20 REPRESENTING SLs Translation prototypically involves getting a text in one language and providing it in another. SLs have no standard written form Problem: State-of-the-art MT systems use the word-forword approach, producing translations that lack of syntax Solution: abstract from the sign synthesis problem by using a writing system Existing writing systems: HamNoSys Stokoe Notation Movement-Hold model Radutzky s Notation AEWLIS

21 Communication Channels The following Communication Channels (CCs) are considered to be relevant in LIS: Hands Direction Body Shoulder Head Facial Labial Gaze. The CCs are assumed to be each other orthogonal, thus requiring an independent recording of the actions performed on each of them. 21

22 The ATLAS Sign set The set of signs used within the ATLAS project includes: Signs associated with Lemmas included in the Radutzky dictionary Signs widely used within the various Italian deaf communities, but not included in the Radutzky dictionary 22

23 Information Tracks A sentence is represented, in AEWLIS, resorting to a set on concurrent synchronized Tracks. The following 4 Tracks (TRs) are used: Lemma: Used to store information items related to the Lemmas, including, among the others, the lemma s ID, Identifier, its Part of Speech, and the set of Modifiers for the Lemma. 23

24 Information Tracks (2) Sentence Structure: Used to store information items related to the Structure of the Sentence, including, among the others, proper pointer to the related syntax tree, the sentence s syntactic and semantic roles, and its speech acts.

25 Information Tracks (3) Channel Modifiers: Its usage is twofold: To record the actual values of the following sign parameters: Sign Spatial Location, for Relocatable signs Action performed by the not-signing hand, when the sign requires just one hand. To record the differences on the various Communication Channels between the actual values of the performed sign and the default ones stored in the ATLAS MultiMedia Archive (AMMA)

26 Information Tracks (4) Time Stamps: Used to store info related to the Time Stamps associated with the sentence Time Slices

27 Oggetto TESTO The ATLAS platform TEXT Com. Attore Virtuale BLOCCHI ESTERNI di TRASFORMAZIONE Da Multi-Media a TESTO TRASFORMAZIONE in COMANDI per ATTORE VIRTUALE

28 AMMA : ATLAS Multi Media Archive ATLAS Multi Media Archive ATLAS DBs 28

29 ATLAS : Possible fall-outs Opportunities for New services E-inclussion New professions All the developed tools made available via open source to the scientific community

30 ATLAS : By-products LIS Dictionary for Smart Phones Annotation tool for Corpora creation Grammar and generator for LIS Service Portal Assisted editor for manual translation