PART I. Signing Books for the Deaf. M14 M18 Different approaches in the production of signing books WP05. RE C. Schmaling, L. Pyfers, J.

Transcription

1 PART I Project Number: TAP 4209 Project Title: Signing Books for the Deaf Deliverable Type: PU Deliverable Number: D05.1 Contractual Date of Delivery: Actual Date of Delivery: Title of Deliverable: Workpackage contributing to the Deliverable: Nature of the Deliverable: Authors: M14 M18 Different approaches in the production of signing books WP05 RE C. Schmaling, L. Pyfers, J. Robinson Abstract: Keywords: Classification: The objective of the Signing Books project is the development of guidelines for the production and distribution of signing books: books in sign language, on video or CD-ROM. In WP3 "State of the Art, signing books from different countries were analysed. In WP5, test material was designed and produced by the partners. The focus was on creating materials which included consideration of those issues identified as of major relevance in which there was substantial variation across existing productions, based on the information collected in WP3. Some test materials were produced specifically for one of the target groups (children, students, adults); other 'general' testing materials were for use with all three target groups. The latter address questions such as colour/contrast, eyeline, signer orientation and number of signers, as well as dealing with translation issues. This deliverable describes the production activities in detail. The material produced in this workpackage will be used for testing in WP6. Signing Books, Sign Language, Deaf Name of Client: FODOK Distribution List: FODOK, City U, Hamburg U, Signum Authorised by: Liesbeth Pyfers Issue: 02 Reference: Del. 5.1 Total Number of Pages: 46 Contact Details: Constanze Schmaling, Institut für Deutsche Gebärdensprache, Universität Hamburg, Binderstr. 34, DE Hamburg, Tel: , Fax: , constanze.schmaling@sign-lang.uni-hamburg.de 1

2 PART I...1 PART II...3 EXECUTIVE SUMMARY...3 SCOPE STATEMENT...4 PART III INTRODUCTION: SUMMARY AND OBJECTIVES OF WP GENERAL TEST MATERIAL Colours Eyeline Editing Translation vs. interpreting TEST MATERIAL FOR CHILDREN: HANS IM GLÜCK: AN EXAMPLE OF INTERNATIONAL COOPERATION Research questions for children Toddlers Pre-readers Early readers Adolescents The test materials Pre-production Script: Translation Casting Costume, make up, props, scenery Production Filming Voice-over and subtitles Editing instructions Editing Hans im Glück videos Hans im Glück CD-ROMs Results: International co-operation TEST MATERIAL FOR STUDENTS: AN INTRODUCTION TO HAMNOSYS Introduction: the idea The script Writing the script Translation of the script: German German Sign Language Filming: deaf hearing team Editing HamNoSys video HamNoSys CD-ROM Subtitles...42 APPENDICES...43 Appendix 1: Technical equipment...43 General test materials: Editing and Eyeline...43 General test materials: Translation vs. interpreting...43 Children's test materials: Hans im Glück...43 Students' test materials: HamNoSys...44 Appendix 2: Timepath for different test materials...44 Timepath general test material: colours...44 Timepath general test material: eyeline...44 Timepath general test material: editing...44 Timepath general test material: translation issues...45 Timepath children s test materials...45 Timepath students' test materials

3 PART II EXECUTIVE SUMMARY Signing Books for the Deaf is a European project funded under the TAP-DE programme (TAP- DE 4209). The project started on January 1, 1998 and will run for 24 months. Partners in the project are: FODOK NL (co-ordinator), City University UK, Hamburg University DE, Signum DE (associated partner), and sdg NL (financial management). Sign language is the preferred language of the early deaf (persons who were born deaf or became deaf early in life) and the only fully accessible language for most of them. There is no generally accepted writing system for sign languages. Therefore, the term signing books is used in this project to indicate books in sign language, on video or CD-ROM, comparable to talking books for the blind. For many early deaf persons, signing books and signed television programmes are the only means of independent and equal access to general information and hearing culture They are also the only means of recording signed information and culture from the Deaf community, for future reference and/or for distribution to a larger audience. In many EU countries, videotapes (and more recently, CD-ROMs and other electronic media) are being produced as an alternative to printed texts for Deaf sign language users. These present text in sign language, often in combination with printed text (captions or text screens) as well as visuals (static or moving pictures) and sound (the original soundtrack or a voice-over). However, these signing books are a relatively new development. In some EU-countries, only few signing books have been produced; in others, as many as signing books may be available. In most EU-countries, these productions are still largely semi-professional or amateur, ad-hoc productions, made by schools for the deaf, organisations for the deaf, or organisations of parents of deaf children. Most of these productions are only marketed on a local or regional level. There is no framework for international or even national co-operation. Organisations in this field (producers, publishers, intermediate users and end-users) are often unaware of development and production activities in other countries. Research on the effectiveness (cost-effectiveness and usability) of productions, as well as on how to improve the quality of signing books, is practically non-existent. The objective of this project is to develop guidelines for the production and distribution of signing books. A first step was to collect information on signing books within a framework developed by the consortium (activities undertaken in WP3). Following this, important issues were selected for which it was decided to develop and produce test materials to use for testing within WP6. The results from the testing will go into the final document: Recommendations for Signing Book Productions. In this workpackage, based on the information gathered and analysed in WP3, different types of test materials were produced: some were designed specifically for one of the target groups (children, students, adults); other 'general testing' materials were for use with all three target groups. The latter address questions such as colour/contrast, eyeline, signer orientation and number of signers, as well as dealing with translation issues. This deliverable describes the production activities in detail. 3

4 SCOPE STATEMENT This document is the output of Workpackage 5 of the Signing Books Project (TAP DE 4209). The activities for this workpackage began on 1 July 1998 and ended on 28 February

5 PART III 1 INTRODUCTION: SUMMARY AND OBJECTIVES OF WP5 In WP3 State of the Art, a large number of signing books from different EU countries were described and analysed within a framework developed by the consortium. As a first step in WP5, topics were selected from this framework as important issues to be addressed in the effectiveness testing of WP6 and which will be covered in the Signing Books guidelines to be published and distributed at the end of the project. After selection of the relevant areas, the development of the test materials itself began. Some 'general' test materials were produced by Signum and City U which can be used with different target groups. These materials address questions such as colours/contrasts, eyeline, signer orientation, number of signers, as well as translation issues. The production of these materials is described in chapter 2. Other test materials were developed and produced specifically for one of the three target groups (children, students, and adults). Different approaches were taken. For the children s test materials, already existing video of a Grimm brothers fairy tale were used in combination with a Dutch Sign Language narrator. For the students, a course book for the Hamburg Notation System (HamNoSys), a transcription system for sign languages, was written and then produced. Both Hans im Glück and HamNoSys were produced by Signum in Germany, the children's materials in cooperation with the Dutch partner. The translation and the filming of the children's materials were done in the Netherlands by the FODOK. In Chapter 3 and 4, the different steps in the production process of these test materials are described in detail. These materials will be tested in WP6 with different signers in Germany, the Netherlands and the UK. Selected materials will be made available for testing on the Internet in order to get additional feedback from other countries. 5

6 2 GENERAL TEST MATERIAL 2.1 Colours For testing colours/contrasts, a video of approximately ten minutes length was produced with different shots of 30 seconds length each. The presenter wears different shirts with varying background colours. He signs three sentences in German Sign Language ( Hello, welcome to the Signing Books project. All over Europe, approximately 200,000 Deaf people use sign language as their first language. Deaf people receive information through sign language.) The text is the beginning of the introduction to the Signing Books project. This signer had been previously filmed signing the complete project introduction at the time of the beginning of the project (and placed on the project website), so rehearsals for this test material were very brief. Filming took place in a studio with two deaf technicians who were responsible for the filming procedure (see also chapter 4). In the studio, seven different backgrounds were available, most of them lighter colours (see the examples on the left side). The other background colours were added on the computer by keying the signer (see the examples on the right side). On the video, the following colours are shown: A. Signer wears grey shirt; background colours: 1.blue; 2. light yellow; 3. light purple; 4. light pink; 5. light green; 6. yellow; 7. dark orange; 8. bordeaux; 9. dark purple; 10. dark blue; 11. dark turqouise; 12. dark green; 13. green; 14. white. B. Signer wears red shirt; background colour: light blue. C. Signer wears light blue shirt; background colours: 1. white; 2. (light) grey; 3. light purple; 4. light yellow; 5. red; 6. red and green. D. Signer wears green-grey shirt; background colour: white with turquoise spots. 6

7 In one of the Hans im Glück videos (1(i)), the signer was also keyed with different background colours: orange, yellow, green, turquoise, blue, pink, white, and grey (some are illustrated in chapter 3.4). 7

8 2.2 Eyeline This section of the test materials is designed to explore viewer preferences relating to the physical arrangement of signer(s) on screen. This area is of interest for a number of reasons, relating to twodimensionality, viewing angle, and the role of eye gaze. The representation of sign languages on screen inevitably involves a two-dimensional presentation of a three-dimensional phenomenon. The resultant flattening of the image can cause some perceptual problems for viewers since information about extent of movement in the plane intersecting at a right angle with the screen cannot easily be seen. In addition, in face-to-face conversation, signers use eye-gaze to maintain discourse and enable turn-taking. The optimal arrangement of two or more signers facing each other and the need for the viewer to see them at an optimal viewing angle are in inherent conflict. Producers are faced with choosing among such options of having signers with their eyeline to camera or to an in-studio audience in- or off-shot. These test materials were designed to explore Deaf people s preferences for eyeline and grouping of signers to find the best balance between these conflicting demands. We are also investigating viewer preferences in relation to gaze. Signers on screen may or may not direct their gaze to the viewer. Since normally, signers monitor viewers comprehension through gaze, arrangements where this does not occur may be less engaging to the viewer. Most of the clips for the eyeline monitoring task have been taken from video materials previously produced at City University in conjunction with CACDP (the Council for the Advancement of Communication with Deaf People). These materials was recorded for the production of a video The Linguistics of British Sign Language, created to accompany a book of the same name by Bencie Woll and Rachel Sutton-Spence (1999). The video comprises introductory material explaining the aims of each chapter, (one person signing to camera) followed by extracts of signers in groups (between 2 and 4) in naturalistic conversation. Filming took place in a variety of indoor and outdoor locations. Each clip is between 30 and 50 seconds in length. The signers appearing in the test materials are all experienced sign language actors who have had experience of video work. Clip 1: The signer is looking directly at the camera: eyeline direct to viewer. Clip 2: The signer is looking at a group of children sitting on the floor, telling a children s story. The eyeline is directed at the children s level. The signer is facing directly forward, with shoulders hunched talking to child. 8

9 Clip 3: The signer is talking to an audience. The eyeline is at adult level. The signer is facing directly forward, but his body moves to address the whole of the audience, from left to right. Clip 4: The signer is talking to another person at adult eye level, but three quarters facing the camera. Clip 5: The signer is talking to another person who is located to the side of the camera, so the eyeline is just off the camera. Clip 6: Two signers are talking to each other, located sideways to each other, with both facing the camera. Clip 7: Two signers talking to each other, orientated so that they are partly facing each other and partly facing the camera Clip 8: Two signers talking, both visible in shot, with one signer facing the camera and signing, and with the other having her back to the camera. 9

10 Clip 9: Two signers on screen, one talking to someone who is offcamera, with the other signer watching from an angle. Clip 10: Two signers talking to each other, 3/4 facing the camera, 1/4 facing each other. Clip 11: Two signers talking, with a third signer slightly out of group. The third person is watching from slightly behind the left of the main signer. Clip 12: Three signers in a circle round a table in conversation. All are looking at each other, rather than at the camera. Two are 1/2 facing the camera, and one person is 3/4 facing the camera. Clip 13: Two signers sitting at right angles, with one signer facing but not looking at the camera. Clip 14: Three signers, two signers 3/4 facing the camera, one signer facing the camera directly. 10

11 Clip 15: Four signers chatting in a circle, one signer talking with back 3/4 to camera. Clip 16: Four signers in a square, chatting round a table. The back two people on the table are 3/4 facing the camera, while the two at front of table are 3/4 facing away from camera. Clip 17: Four signers, two signers facing camera, two signers side on to the camera. Clip 18: Four signers in a row facing camera full on. Clip 19: Four signers standing in a circle, two with back to camera. The two signers facing the camera, their eye gaze is watching the two signers with back to camera. Clip 20: Split screen: Signers filmed separately in 7/8 view and then the two images were combined. 11

12 2.3 Editing This section of the test materials is designed to explore viewer preferences in relation to a variety of editing techniques. This is a topic which needs specific attention in the context of Deaf viewer preferences for two reasons. Firstly, because the message content is carried solely in the visual modality, editing between shots must be carefully done to ensure that information is not lost in the edit. Unlike filming of spoken language materials, a shot cannot be changed in the middle of a signed sentence, and the person signing must be seen on-screen throughout his or her turn. Secondly, because there is no written form for sign languages in widespread use, presenters cannot take cues from, e.g. an autocue in sign language. Therefore, it is often necessary for presenters to memorise sections of materials (see the discussion of other test materials). This in turn means that there must be frequent edits between memorised chunks, again emphasising the importance of obtaining feedback on the most preferred forms. We chose a variety of clips taken from the Sign Linguistics video. A variety of within-turn edits had already been used in the video during the introductory sections for each chapter. Where necessary, additional footage was shot of the same Deaf presenter and other edits implemented with a variety of morphing techniques. Clip 1a: Hardcut editing: Presenter talking with another signer. Clip 2a: Fade: Fade into presenter signing. Clip 3a: Page peel: From chapter title to page peel of presenter. Clip 4a: Flip editing: Presenter signing sentence, then flip to reveal her signing another sentence. Clip 5a: Wipe: Presenter signs sentence followed by a wipe, and then signs another sentence. 12

13 2.4 Translation vs. interpreting As well as material authored directly in a sign language, many signing books use material originally prepared in spoken language for a general audience. This material then requires translation and insertion into the video image. These test materials were designed to answer questions about whether live, simultaneous interpreting, or prepared translation is preferred for material originally created in written/spoken language. An experimental test video was developed to deal with this question. A script was devised, based on a magazine article about Simon, the virtual signer. The content relates to an animated signing system currently in development which transliterates from written text (in English) to a signing avatar. This topic was chosen as one likely to be of interest to test subjects and also posing complexities in translation. A hearing interpreter with extensive experience of broadcast interpreting from English to BSL, and an experienced Deaf television presenter were recruited. The following test materials were created: Sample 1: Both the hearing signer (1a) and the Deaf signer (1b) interpret the script into BSL from an autocue with the script in written English, without previously seeing that section of the script. Sample 2: The hearing signer interprets while listening to an audio recording of the English script, without having heard or seen that section of the script before. Sample 3: Both the hearing signer (3a) and the deaf signer (3b) interpret a section of the script from an autocue, each having had time to read the script, plan and prepare the interpretation. Sample 4: The hearing signer interprets while listening to a previously heard section of the audio recording of the script. Sample 5: The hearing signer (5a) and the Deaf signer (5b) each prepare a translation of a section of the script and then are recorded signing that section from memory. Sample 6: The hearing signer (6a) and the Deaf signer (6b) are each recorded signing a section of the script using the cueing method of their choice. 13

14 3 TEST MATERIAL FOR CHILDREN: Hans im Glück: An example of international cooperation 3.1 Research questions for children One the conclusions of WP3 was that there is very little research available on requirements for, and usability of, signing books for children. It was therefore decided that in WP6 data would be collected on a number of basic issues related to the attractiveness and effectiveness of signing books for deaf children, as well as the cost effectiveness of productions and production methods. It was also decided, on the basis of the results of WP3, to distinguish four age groups within the child population, based both on the age of the children as well as on the primary objectives that producers/publishers address for each age group Toddlers 2-4 year olds: The primary objective is to teach sign language to children as well as to their (mostly) hearing relatives. These videos are based on, and often show on camera, alphabet books with pictures and words. When reading books with children in this age group, the interaction between the reader and the child is of most importance. The reader adapts his/her reading to the age, language level, and interests of the child. Books are read over and over, and each time, the story, and the interaction about the story, is slightly different. Alphabet books on video for deaf children, on the other hand, do not allow much interaction, and are most likely watched by parent and child more passively. A CD-ROM however, does allow for interaction between reader, child and book, and might be more suited to the objectives for this age group. It was therefore decided to develop test material of a signing book for toddlers to compare the attractiveness and effectiveness of the video medium relative to CD-ROM for this specific age group Pre-readers 4-8 year olds: The primary objective is to introduce children to stories and literature from the hearing community, and provide other pre-reading activities. Signing books for this age group are mostly hearing books, translated and presented in sign language, illustrated by a range of visualisation techniques: pictures from the book, and/or scenes performed by actors. Signing books for this age group vary from videos that show a story presented in sign language without any pictures or other visual support for the content of the story, to stories presented in sign language and performed by actors. The majority of these signing books show the story signed by a presenter, signing or standing, with static pictures presented either simultaneously with the story (picture-in-picture, or Chroma key), or with the story and static pictures presented sequentially (part of the story is told by the presenter who is shown full screen, then one or more pictures are shown full screen, and then the presenter continues with the story). Most signing books have a voice-over, to make the presentation accessible to hearing viewers with limited or no fluency in sign language. Many of the differences between signing books for this age group are directly dependent on the source materials: the quality of the story, the quality of the pictures, and the relationship between the signing of the text and the pictures (from hardly any relationship to almost complete redundancy). More general issues to be addressed for this age group, however, are the need for visual support of stories in sign language; the type of visual support (still images or dramatised scenes) that is most effective, and the optimal way to combine visuals and story. 14

15 3.1.3 Early readers 8-12 year old: Many signing books for this age group are based on written language books and stories as well. Since books for this age group generally have few or no pictures, a limited amount of pictures or other visuals appear in the signing book productions, although in some cases, pictures or other visuals are made specifically for the sign language production. In the Netherlands, a number of regular films for children have been made accessible to deaf children of this age group, by adding one or more sign language interpreters, since it is generally assumed that the reading level of deaf children of this age group is insufficient for them to benefit from subtitles. The research questions for this age group therefore partially overlap with those for the 4-7 years olds: need for visual support, type of visual support, and the optimal way to combine visuals and story. Relevant also for this age group is the question of subtitles: Are children in this age group able to benefit from subtitles if they are combined with a story presented in sign language? Will subtitled signing help children learn to read subtitles, or will subtitles distract them from the sign language? As the amount of visual information presented on the screen increases, subtitles may become more difficult to read, i.e. it may be that subtitles are effective for stories presented in sign language or where there is a sequential presentation of sign language and visuals, but they may not be effective with a simultaneous presentation of sign language and visuals. Finally, it may be that subtitles are effective for stories presented on CD-ROM, where the child can read or re-read the subtitles at his or her own pace, but they may not be effective when used on videotape with this age group Adolescents year olds: Signing books for this age group are generally not so much intended to support the reading of books, but rather to replace books. Signing books for this age group are most often educational or informational videos, either developed specifically for deaf teenagers, or are translations from productions made for their hearing peers. They address topics relevant for both hearing and deaf young people, such as drugs and sex education, and also topics of specific interest to deaf young people, such as Deaf culture, Deaf identity, Deaf role models, etc. Productions made for hearing young people are made accessible by adding subtitles, by adding sign language interpretation, or by adding both. Research issues for this age group include the best way to present information to deaf youngsters in sign language on video, the specifics of sign language / video literacy as compared to spoken language / print literacy, and the effectiveness and attractiveness of signing books made specifically for deaf viewers, versus hearing programs that have been made accessible by adding subtitles and/or a sign language interpreter. 3.2 The test materials Limited resources were available for the development of the test materials. It was therefore decided that in WP5, only one set of test materials would be developed for children, partly using existing material copyright to Signum (an associated project partner). The existing material was the videotape Hans im Glück, a Grimm brothers fairy tale, presented in German Sign Language by Thomas Bauer, and visualised through performance by actors. Fairy tales were seen as being of interest for children of different ages; the test materials would therefore be acceptable for at least three of the four age groups. For toddlers, a Dutch multimedia publisher agreed to make an interactive story for 2-4 year old hearing children (Dick Bruna: Nijntje en de dieren) accessible to deaf children in this age group by 15

16 adding a sign language translation. Funding for the production of the sign language translation is being sought outside the project budget. For the year olds, an informational video made by Kuurojen Liitto Ry (the Finish Deaf Association) was selected to compare the effectiveness/attractiveness of an adapted version of a foreign Deaf-made production with an adapted version of a production originally made for hearing young people in young signers own country. The Finnish video will be subtitled in Dutch to make it accessible to Dutch young people; additional funding is being sought for the translation of the Finnish voice-over into Dutch, and for adding subtitles to the video. Both of these videos will be produced for publication in the Netherlands. For WP5, Signum developed test material in order to collect data on the following issues: Is it necessary for deaf children (of different age groups) to have (parts of) a story visualised, or is it sufficient to have a story presented in sign language without accompanying visuals? If visualisation is necessary or preferable, are still pictures or graphics then sufficient, or should (parts of) the story actually be dramatised? If visualisation of the story is to be preferred, should the sign language and the visuals then be shown sequentially, or are they best presented simultaneously? If the sign language and the visuals are best presented simultaneously, what is the best way to combine visuals and sign language: sign language in the main screen, with visuals shown in a box or bubble? or vice versa; using Chroma key? or a split screen presentation? If boxes or bubbles are used, what is the preferred size, location and colour; should they blend with the background or contrast? What age groups benefit from subtitles and what is the best way to add subtitles? For what age groups is a voice-over necessary or preferable? What are the advantages and disadvantages of VHS vs. CD-ROM for different age groups? What are the advantages and disadvantages of translating a foreign signing book by adding subtitles as compared to developing a national production? What are the advantages and disadvantages of recording sign language in one country while having the editing done by the producer of the original production in another country? To research these issues, the following Hans im Glück test versions were specified: 1. Story presented in SLN, no visuals 2. Story presented in SLN with static pictures taken from dramatised scenes 2a: SLN and static pictures shown sequentially (SLN or picture in the main screen) 2b: SLN and static pictures shown simultaneously, using Chroma key 3. Story presented in SLN, with dramatised scenes from Hans im Glück 3a: SLN and scenes shown sequentially (SLN or scenes in the main screen) 3b: SLN and scenes shown simultaneously, using Chroma key: SLN in main screen, scenes in background 3c: SLN and scenes shown simultaneously, using picture in picture technology: scenes in main screen, signer in a box (different sizes, locations, different colour background) 4. With/without subtitles: 16

17 If possible, all materials presented in interactive format (on CD-ROM), with the user being able to select a version with or without subtitles. Or: 4a: type 1, with subtitles 4b: type 2b, with subtitles 4c: type 3b, with subtitles 3.3 Pre-production Script: Translation The German production of Hans im Glück was based on the original text of the Grimm brothers story, as published in In the DGS (German Sign Language) version, a deaf presenter freely translates the text. The German voice-over is a direct back-translation of the DGS version. For the translation from German into Sign Language of the Netherlands (SLN), it was decided not to use the original German text published in 1857, but to translate the DGS/German voice-over from the video into easy-to-read Dutch. The translation from DGS into written Dutch took approximately four hours. In the translation, the story was divided into the same ten scenes as in the DGS version. The Dutch text was then translated into SLN by a translation team comprising one hearing and one deaf person. Both were first-language users of SLN and are experienced SLN translators. The translation was based on the written Dutch text as well as the DGS video. They watched the DGS video about 15 times in order to prepare the translation, which took in all about five to six hours. They reported that having the DGS video available was helpful for comparison and inspiration Casting Since the time available for translation and recording was very limited, it was decided that the team responsible for the translation would also undertake the video presentation. Since the main character in the story is a man, and since the German version has a male presenter, it was decided that the Dutch version should also use a man as presenter. The hearing member of the translation team is a man and has also been involved in the production of several SLN videos, and he was therefore asked to present the story to camera. A third person was added to the team to handle make-up, costumes, and act as the production assistant. The presenter and production assistant both had participated in a Signing Books workshop organised by FODOK in autumn 1998, and were therefore familiar with the project and the many issues the project is addressing. The presenter used an audio recording he had previously made of the script to cue himself while practising the SLN presentation. The audio recording was also used during the video recording. Practice of the presentation took approximately 8-10 hours. In order that the voice-over matches the presenter as closely as possible, the presenter did the voiceover himself Costume, make up, props, scenery A simple costume was selected: dark clothing and a flat cap, comparable to the cap Hans wears in the dramatised scenes, since the video recordings in SLN were to be used in many different ways, from full screen presentation, to a small picture-in-picture format. The cap was also used as a 17

18 visual cue for the SLN name sign for Hans (CAP). No other props were used. The presenters hands and face were made-up, so that these would look natural in artificial studio lighting. 3.4 Production Filming Filming took place at the Effatha School for the Deaf in Zoetermeer, the Netherlands. The recording was done in an improvised studio in the sports hall in front of a rented portable Chroma key blue screen, on U-Matic tape. The filming was done on 18 December 1999, by a team consisting of the presenter, the sign language consultant, the production assistant, a cameraman, and a representative of the Signing Books project. The recording procedure was as follows: 1. All watched the story in DGS (the original video version) on a monitor with a VHS-deck. 2. The cameraman checked lights and camera. 3. The presenter removed his glasses, checked his location and orientation. 4. The production assistant started the studio recording of the text for that scene. 5. The cameraman started the camera. 6. The scene and take-numbers were recorded on a board. 7. The presenter began to sign. 8. The SLN consultant stood next to the camera to cue and monitor the presenter s signing. 9. After recording each scene, all watched the recording and checked for cut-offs, mistakes, etc. 10. If the scene was found to be satisfactory, preparations for the next scene were started. Overview of the general set up during the filming. 18

19 The presenter checks the location of his feet marked with clear tape on the floor. The sign language consultant stands next to the camera to cue and monitor the presenter s signing. Filming of the entire story (15 minutes in length) including preparation, discussions, etc. took approximately four hours (the tale indicates the number of takes needed per scene). After filming, the U-Matic tapes were copied onto VHS, and the originals were sent to Hamburg for editing. Scene number Scene length (in Number of takes needed takes (warm-up, reflection of lights on glasses / lights / forgotten to introduce name / expression (too many cut-offs) Average (excluding scene 1: 2.7) 19

20 It was possible to record fairly long sequences, because rather than memorising the text, the presenter used the audio recording for cueing. Several scenes had to be recorded more than once, either because a wrong sign was used (e.g. a regional variant instead of a more generally used sign), or because the recording was found to be of unsatisfactory quality (cut-offs, shadows, etc.) Voice-over and subtitles The voice-over was recorded after the story was filmed, using the (unedited) video recording of the SLN presentation. The final recording of the voice-over was made in a sound-proof booth at Effatha School on DAT-tape. During the final recording, the videotape was used to pace the voiceover. The length of the audio recording therefore matches the length of the filmed scenes. This, however, had the disadvantage that the microphone also recorded the background hum of the video recorder. For the voice-over, the original printed text was used rather than a back-translation from the SLN. The same text was used as a basis for the subtitles Editing instructions A VHS tape with all filmed scenes accompanied the detailed instructions for the editor in Hamburg. Examples of these are illustrated in the tables below. Version 1: no visuals Scene Beginning End Hello 00:41: :41: :07: :08: :11: :13: :13: :14:15.12 Version 2a: narrator + stills, sequential presentation Scene Beginning End Stills Hello 00:41: :41: Hans waves 1 00:07: :08: Hans waves 1.2 gets gold 1.3 gold on shoulder 1.4 sees horse 2 00:11: :13: sees horse 2.1 trades gold horse 2.2 rides away 3 00:13: :14: rides away 3.1 Hans on ground 3.2 Horse... Version 2b: narrator + stills, simultaneous presentation Scene Beginning End Still, starting point * Hello 00:41: :41: Hans waves: 00:41:

21 1 00:07: :08: gets gold: 00:08: gold on shoulder: 00:08: :11: :13: sees horse: 00:08: trades gold horse: 00:12: :13: :14: rides away: 00:13: Hans on ground: 00:14: Horse: 00:14: Starting point: preferably at the beginning of a new sentence ending point = next starting point Version 3a: narrator + clips, sequential presentation (as in the original Hans im Glück video) Scene Beginning End Clips Hello 00:41: :41:26.20 Hans waving 1 00:07: :08: : Hans gets gold 2 00:11: :13: : trades gold for horse 3 00:13: :14: : gets thrown by horse... Version 3b: narrator + clips, simultaneous presentation Scene Beginning End Clip anchor point* Hello 00:41: :41:26.20 Hello 1 00:07: :08: anchor point: receiving of gold 2 00:11: :13:12.07 (SLN: 00:08:17.00) 2: anchor point: trade over gold horse 3 00:13: :14:15.12 (SLN: 00:13:00.00) 3. anchor point: Hans on the ground (SLN: 00:14:12.20)... There are no time codes for the drama clips. The length does not always correspond to the length of the narrative, being either too short or too long. In these cases, the beginning and ending of the drama clips is adapted (shortened or frozen) to make them fit better. The actual beginning and end points of the drama clips can be altered in most cases. It was decided, however, that the central action in each drama clip be shown at the same time as the narration. In the table, this central action is called the anchor point. The narrator and the drama are linked at the anchor points, the beginning and end points of the drama clips are then modified to make them fit the length of the signing. 3.5 Editing All video versions of Hans im Glück were edited into ten scenes, following the original German Sign Language video which consists of twenty scenes with the sign language presenter alternating with ten drama scenes. Several of the original drama scenes were slightly re-edited for the Dutch version of Hans im Glück (for example, where Hans moves out of the picture in a different direction) but generally, they were identical with the original. For the stills, suggestions from the Dutch partner were followed but one or two stills were added in each scene to improve understanding. 21

22 3.5.1 Hans im Glück videos Eight different versions of the video were created for testing. Their structure is illustrated below: 22

23 Version 1 (i): no visuals/blue screen The story is signed with no visuals. In all scenes, the SLN narrator is shown as in the original with a blue screen background. Version 1 (ii): no visuals/various background colours The story is signed with no visuals. In scenes 1-8, the narrator is keyed with various background colours. Scene 1: orange Scene 2: yellow Scene 3: green Scene 4: turquoise Scene 5: blue Scene 6: pink Scene 7: white Scene 8: grey Scene 9-10: original There are scene-change effects between the last four scenes: 7-8: peel-wipe left-right 8-9: peel-wipe right-left 9-10: fade out - fade in Version 2a (i): narrator + stills, sequential/blue screen The narrator and the stills appear sequentially. A series of stills after each signed sequence (usually between 3 and 5) re-narrate what has been signed. The presenter is shown on the original blue background. The stills have different lengths: Scenes 1-7: stills each 2 secs. long Scenes 8-10: stills each 3 secs. long 23

24 Version 2a (ii): narrator + stills, sequential/background varies Narrator and stills are shown sequentially. The narrator background varies, and both stills and narrator fade in and out. Scenes 1-4: The last still of the scene remains as background for the following signed part, but is much brighter. Scenes 5-10: A still of the forest (very bright colours) is shown as background for the signer. Version 2b: narrator + stills, simultaneous Narrator and stills appear simultaneously. Different effects have been selected which are used in two scenes each: Scenes 1-2: Split screen: narrator left, stills right; separated by a thin black border. Scenes 3-4: Split screen: narrator right, stills left, with a soft border. Scenes 5-6: Stills are shown in a small box (with a black border) in the top right. Scenes 7-8: The narrator is in a mediumsize circle (with an olive border) in the lower left. 24

25 Scenes 9-10: The narrator is keyed: between the stills a page flicking -effect has been added. Version 3a: narrator + videoclips, sequential As in version 2a (i), narrator and visuals alternate. Here, the videoclips from the original Hans im Glück video are used. The presenter is shown on the original blue background. 25

26 Version 3b (i): narrator + videoclips, simultaneous/narrator in box Narrator and videoclips seen simultaneously. At the end, and occasionally at the beginning of scenes, freeze-frames of videoclips extend to the end of the signed section. Scenes 1-4: The narrator is keyed at different locations. Scenes 5-9: The narrator is in a box (with blue background) of varying sizes, borders, and locations. Scenes 5-6: Narrator in a small box with a green border in the lower right. Scene 7: Narrator in a small box without border in the lower right. Scene 8: Narrator in a medium-sized box, with soft border, in the lower right. Scene 9: Narrator in a medium-sized box, with a still of the fountain as background. In the first half of the scene, the box is borderless and is located in the lower right; in the second half, the box has an olive border and is located in the lower left. Scene 10: Narrator in a circle with blue background and green border in the lower right; the circle moves to the lower left during the scene. 26

27 Version 3b (ii): narrator + videoclips, simultaneous/split screen, clips in box Narrator and videoclips are shown simultaneously. At the end or beginning of scenes, video clips are frozen and extend until the end of the signed section. Scenes 1-3: Split screen: The narrator is on the left and the video clips are on the right, separated by a thin black border. Scene 4-6: Split screen: Narrator on the right, videoclips on the left, separated by a soft border. Scenes 7-8: Videoclips in a small box with a black border in the top right. Scene 9: Videoclips in a mediumsized box with a soft border in the top right. Scene 10: Videoclips in a small box with a thin olive border at the bottom centre. The visuals are slightly transparent so that the signer s hands are visible. 27

28 Videos: Post-production The videotape received from Hamburg was re-edited in the Netherlands before being distributed to parents of deaf children and to schools for the deaf in the Netherlands. In the parents videotape the order of the various versions was changed and scene numbers were added to the video in order to facilitate the completion of the accompanying questionnaire. For the deaf schools, a videotape was prepared which included not only samples from the Hans im Glück test materials but also extracts from other children s videos Hans im Glück CD-ROMs Five different CD-ROM versions were produced which all make use of the Hans im Glück video materials. Three are produced with subtitles, two as interactive material Subtitles For testing, the CD-ROM versions of the three video versions 1 (i), 2b, and 3b (ii) were subtitled (see also 3.4.2). On the CD-ROMs, two different subtitling formats were used: On the CD-ROMs with no visuals (1i) and with narrator and stills simultaneously (2b), the subtitles appear within the video-frame (as it would be if a video were subtitled). The background blends over the video, and the text appears in black. 28

29 CD-ROM with subtitles in the video frame On the CD-ROM with simultaneous narrator and drama clips (3bii), the subtitles are shown below the video frame, an option not available with videotape. The text is black on a grey background. 29

30 CD-ROM with subtitles below the video frame In all three versions, the user can move from one scene to the next (and back) by using the forward and back buttons respectively Interactive CD-ROMs Two other CD-ROMs of Hans im Glück can be used as interactive material. One has an index at the beginning of the CD which consists of a still from each scene plus a short text which describes the most important event of the scene. 30

31 From this page the user can access each scene which then appears as a new page. From the scenes themselves one can either move on to the next scene (or back see above) or return to the menu. 31

32 The other interactive CD-ROM has a divided screen with the index on the left side (this time consisting only of a still) and the scenes on the right side. The user can access each scene via the index or move from one scene to the next directly. 32

33 3.6 Results: International co-operation International co-operation saves time with regard to the filming of drama scenes for which actors and a filming team are needed for several days. In relation to editing across country borders, several issues are very important, including among many others: The use of compatible technical equipment by partners. Highly specific instructions for the editor e.g.: exact time codes for editing the video; this is even more important if the editor is not familiar with the sign language used by the presenter; specific subtitling instructions. During the editing, some of the special effects requested by the Dutch partners could not be achieved, or achieved only partially for several reasons: The original Hans im Glück video was planned as a sequential production, i.e. the presenter narrates the story in brief sections, interspersed with drama clips after each scene. In the original video, there were no special effects such as simultaneous presentation of a narrator and visuals through keying, inserting boxes or splitting the screen. For the editing of the Dutch version this meant freezing video frames at the beginning or end of some scenes because the narrated text was longer than the drama clips. 33

34 In the original video, the character of Hans returns home to the fountain where the sign language narrator sits. This was not possible in the Dutch version since the background (the fountain) was not available. A substitute scene therefore had to be created. In planning a co-operative production, the following issues should be considered: To show drama clips and narrator simultaneously, it is necessary to plan the drama sections to match the length of the signed sections. To include effects such as split screen or boxes, information needs to be available in advance: the location of the signer and the visuals, and the size and location of the box. The signer needs to be filmed in the location in which he or she will appear on the screen so that the visuals can be filmed so as to avoid information loss. When using a blue screen, it is important that it is large enough. Otherwise problems may be encountered during editing. Special attention also needs to be directed to lighting and clothing colours in order to avoid flicker when keying the signer. 34

35 4 TEST MATERIAL FOR STUDENTS: An introduction to HamNoSys 4.1 Introduction: the idea HamNoSys is a notation system for sign languages which was developed at the Institute of German Sign Language at Hamburg University in The system is mainly used within the context of sign language linguistics, for example to transcribe signs in dictionaries. Deaf students and staff at the Institute who are involved in sign language research have pointed out that a description of HamNosSys is available only in written form but not in sign language. Thus the idea of producing material for HamNoSys in sign language was developed as an example of genuine student material. In HamNoSys, the four manual parameters handshape, orientation, location, and movement are transcribed in linear order. In addition to these, non-manual parameters can be notated. A complete introduction to the transcription system, including all four manual parameters, would have required a video of several hours length. It was therefore decided to concentrate on one of the four parameters orientation which creates most problems for students/researchers who learn and use HamNoSys. Some of the students' material in Germany is available on CD-ROM. Through discussions with Deaf students and members of staff at the Institute it became obvious that many of them found CD-ROMs to be a more appropriate medium for students' materials because of different navigation options. It was therefore decided to produce both a video- as well as a CD-ROM-version of the HamNoSys introduction to test which medium test persons preferred for such materials. 4.2 The script Writing the script The script was written by a hearing person in co-operation with two deaf members of the Institute who have been teaching introductory courses in HamNoSys in different European countries including Germany for several years. In most cases, the written text was modified in line with comments by deaf staff relating primarily to content issues: how and in which order information should be presented. It was decided to begin with a general introduction to transcription/notation systems for sign languages and give a brief introduction to three of them: Bébian, Stokoe, and Sutton. There follows a general introduction to HamNoSys, a presentation of the orientation parameter, and finally a number of examples and exercises in question and answer format. The script not only included the text to be recorded in sign language but also notes on production issues such as background colour, camera position, and signer position. 35

36 III. Kapitel: Einführung in Handstellungen von HamNoSys. Die unterschiedlichen Fingeransatzrichtungen und Handflächenorientierungen werden vorgestellt. Anhand von Gebärdenbeispielen sollen die Fingeransatzrichtung und Handflächenorientierung erklärt werden. 8. Szene: Handstellungen: Hintergrund: Königsblau Aufnahme: von vorne Oberkörper Position von Erzählerin: auf der rechten Seite Einblendungen auf linker Seite: (1) Fingeransatzrichtung (2) Handflächenorientierung (3) HS --> FA/HFO Grafik ERZÄHLERIN Die Handstellungen in HamNoSys setzen sich aus Fingeransatzrichtung und Handflächenorientierung zusammen. Die Handflächenorientierung ist nur in Kombination mit Fingeransatzrichtung möglich. Es gibt 18 Symbole für die Fingeransatzrichtung. Für die Handflächenorientierung gibt es 8 Symbole. Zusammen stehen für Handstellungen 26 Symbole zur Verfügung, die sich unterschiedlich kombinieren lassen. (Example of script) Some scenes of the script had to be rewritten for the CD-ROM version (see below) Translation of the script: German German Sign Language The final version of the script was translated into sign language by the two deaf team members. This took a considerable amount of time. They had to translate the written text into sign glosses as well as plan how best to present the information in sign language so that deaf people without any background knowledge about transcription systems would be able to follow the content. The translation process was regarded by the presenters/translators as a very difficult task. They would have preferred writing the script in glosses initially, rather than translating a script from written German. 4.3 Filming: deaf hearing team Filming took place in a video studio with a deaf/hearing team. The (hearing) studio manager supervised the scene preparation and lighting arrangements. Two deaf technicians were responsible for the actual filming which took an entire week. They prepared the studio each morning and ensured continuity in lighting, etc. The two deaf people who had assisted in the preparation of the script acted as presenters. This was done for a number of reasons. Firstly, they were well acquainted with HamNoSys and had already taught HamNoSys courses, so they already knew the relevant terminology and were familiar with 36

37 it. Having been involved in the script-writing process also helped them to be familiar with at least on a basic level the structure of the script, film content and approach. For the video, 48 scenes were required. Of these 22 consisted of only one (repeated) sign example. The other scenes varied in length, each consisting of approximately five to ten signed sentences. Seven scenes were recorded separately for the video and CD-ROM versions. In the video version, the signer explains the order in which information will be presented. For the CD-ROM this is not necessary since users can decide the viewing order for themselves. Filming was done in front of a blue screen. In most scenes, the presenters signed directly to camera, although sometime the position varied, i.e. to the right or left side of the screen. One scene was shot using a bird s eye view. For this effect, substantial technical effort was required to make filming in front of the blue screen possible. Both presenters are non-professionals, so cueing each other was something relatively unfamiliar. Most scenes had to be shot four or five times because it was difficult for them to remember the text fully and minor mistakes crept in. Communication between hearing and deaf members of the team was also occasionally difficult. 4.4 Editing Editing took place in several stages. The first step was to select the scenes to be used for the video. Exact time codes were given to the editor. Apart from the seven scenes which differed for the CD- ROM and video, all scenes were used for both versions. While editing the video problems arose: Some scenes had been filmed in slightly different lighting, with the signer(s) at varying distances from the camera, or in different camera positions. This made it difficult to paste sections together, and sometimes a black or white screen or wipe-effects were inserted to make these differences less obvious. Issues like these should preferably be dealt with during the filming to avoid problems in the editing process. 37

38 4.4.1 HamNoSys video All scenes were filmed in front of a Chroma key blue screen. Different kinds of visuals were inserted afterwards next to the signer, these included: text only symbols and text picture and symbols In the exercise section, the signer is keyed in front of a purple background. sign example correct answer 38

39 4.4.2 HamNoSys CD-ROM As the goal of this production was a testing base, rather than a finished multimedia production, HTML (World Wide Web page formatting language) was chosen instead of proprietary tools such as Macromedia Director. It was assumed that users of the HamNoSys test material would have a degree of familiarity with the user interface conventions of the WWW and therefore would have no user interface technical problems in dealing with the test material. For the CD-ROM version of HamNoSys some scenes had to be altered because of the interactive options the medium offers, e.g. in some of the video-scenes, the signer points at different points in space later text insertions were added. Some of this was not necessary for the CD-ROM since one has the index and can choose where to go next. The pages of the CD-ROM are split, with the index on the left side and the films of each scene on the right side of the screen. The user can either go to each scene from the index or move from one scene to the next via the 'forward' and 'back' buttons below the films. Some of the chapters have sub-chapters. These become visible in the index once a chapter is opened: the sub-index then opens as well. 39

40 In the 'exercise section', the answer options appear on the left side of the screen to correspond with the signer s pointing to the left when explaining how the exercise section works. The user can click on an answer and then a clip with feedback on the correctness of the response ( very good or sorry, your answer is wrong ) appears. The user can then choose to go back to the exercise and view the sign again, try another answer or go directly to the next exercise. 40

41 In one scene two text insertions were overlaid to give the user another navigation option. 41

42 4.4.3 Subtitles Subtitles were needed for testing in some scenes. For the testing, one of the sign language interpreters at the Institute was asked to back-translate three scenes (with varying effects/visuals) of the signed text into spoken German. The translation was cross-checked with the sign language presenters. The subtitles appear below the signer within the video frame. The background blends over the video, and the text appears in black. 42