A Limited Communication Domain Mobile Aid for a Deaf patient at the Pharmacy

Transcription

1 A Limited Communication Domain Mobile Aid for a Deaf patient at the Pharmacy Michael B. Motlhabi Student I.D: A proposal submitted for the degree of Magister Scientiae Department of Computer Science, University of the Western Cape Thesis advisor: Dr William D. Tucker June 5, 2012 Abstract This study explores how to implement a health communication aid in a limited domain using mobile phones to support a Deaf person visiting a public hospital pharmacy. The aim is to prevent problems of non-compliance to treatment due to poor communication between a Deaf patient and a pharmacist. The risk of miscommunication is high for Deaf patients in South Africa who communicate in South African Sign Language since most government pharmacists cannot communicate in South African Sign Language. Our study concentrates on limited domain communication with pharmacists because the dialogue is simple to predict and easy to map down. Deaf people with a capital D are different from deaf or hard of hearing as they primarily use sign language to communicate, Deaf people use South African Sign Language as their first language. Deaf users would like a system that can ease the communication between them and the pharmacist during medicine dispensing. A prototype will be developed for a mobile phone using stored sign language videos arranged in a programmatically structured sequence to allow pharmacists to explain medical instructions clearly to a Deaf patient depending on the prescription from the doctor. The technical goal is to implement the prototype on a mobile device and tie the resulting communication structure that will take place between the pharmacist and the Deaf patient to a string of sign language videos a Deaf patient can later watch in order to take their medicine correctly as prescribed by the doctor. i

2 Key words K.4.2 [Social Issues]: Assistive technologies for persons with disabilities D.2.13 [Reusable Software]: Domain engineering D.2.11 [Software Architectures]: Domain-specific Architectures D.4.2 [Storage Management]: Secondary Storage (memory card) ii

3 Introduction SignSupport is an interactive mobile communication aid that operates in a restricted domain to assist in dispensing medication at a public hospital pharmacy to a Deaf patient. The need for concise and clear communication between the pharmacist and the Deaf patient is extremely important for the treatment protocol. This project concerns the definition and construction of the communication flow in a limited domain between the pharmacist and the Deaf patient. Deaf people in South Africa use services like Short Messaging Services (SMS), MMS (Multi-Media Service), WhatsApp, Mxit, , chat (such as Google talk, Skype, etc), fax, video telephony and voice/tty relay services to communicate with each other and with hearing people [23]. Deaf people with a capital D are different from deaf or hard of hearing as they primarily use sign language to communicate [29][35]. Deaf people in South Africa often have problems communicating with hearing people because they use sign language and not spoken language. It is a frequent misunderstanding that signed language is a signed form of a written/spoken language. This is not true, and many Deaf people have low written/spoken language literacy thus reading, writing and lip-reading are not viable options for them [2]. Using a sign language interpreter is expensive because registered interpreters are very scarce in South Africa. Deaf people in South Africa use South African Sign Language (SASL) as a mother tongue and prefer to use it whenever possible. SASL is peculiar in that it spans ethnicities, and represents a culture within South Africa outside the traditional cultures associated with the eleven official languages. In the process of researching the communication structure in a limited domain we will develop a tool called SignSupport to evaluate the communication flow. SignSupport version 3 s interface was designed by Chininthorn from Tu Delft. She moved the project from the doctor to concentrate on the pharmacy context [6]. She designed the front-end interface as part of her Masters project, concentrating on how Deaf people interact with cellphones and studying conversations between pharmacists and Deaf patients. From her work we aim to investigate the subject of limited communication exchange and craft a solution that can be implemented on a mobile phone for Deaf users to help them understand instructions from a pharmacist during medication dispensing. With this interactive system, a Deaf patient can view information on how to take medicines via SASL videos. The scenario is as follows; a Deaf patient visits a public hospital pharmacy with a mobile phone and hands it to the pharmacist (who already 1

4 knows about the application) with a prescription from the doctor. The pharmacist uses the application on the mobile phone to instruct the Deaf patient on how to take the medication correctly. A mobile phone will be used because it is already a common tool of communication among Deaf people and related development has already been done with these devices. The key idea is to use the mobile phone to display videos containing sign language. Deaf patients will not have to rely on an interpreter being present during medicine dispensing. The mobile system thus helps a Deaf user communicate with a pharmacist, and vice versa, without sign language translation. For example, a Deaf patient is asked about their background information via SASL video on the phone and the pharmacist can read the answers from the patient in the form of text (see Table 4 and 5). All of the possible video messages are stored on the phone s memory card to keep costs down. However, if a communication break down occurs, we provide a video relay service to support an interpreted conversation with a remote SASL interpreter or hearing family member who can sign. We plan on using Skype for video relay to clarify issues not covered by canned interactions. We do not utilize automatic natural speech or sign language recognition because these technologies cannot currently guarantee enough accuracy for medical instructions. This prototype is therefore not an expert system and does not use artificial intelligence. The user trials of SignSupport version 3 will be done with a non-governmental organization (NGO) called Deaf Community of Cape Town (DCCT). There the DCCT staff members help trial Information and Communication Technologies for the Deaf (ICT4D) prototypes and at UWC School of Pharmacy with pharmacists testing the pharmacist s side of the prototype. 2

5 Background In the past ten years or so the University of the Western Cape, Computer Science department has been involved in researching and developing communication tools on mobile phones for the Deaf Community of Cape Town (DCCT), an non governmental organisation (NGO). We have researched areas like browser-based and mobile video communication services for Deaf people to find one that Deaf people would like to use in their day-to-day life [35], and approaches to designing and developing a asynchronous video communication tool that adapted synchronous video codecs to store-and-forward video delivery [15]. This work however stems from Looijesteijn, an industrial design-engineering student, designed SignSupport version 1, a mock-up that ran on a PC. A computer science student Mutemwa took the idea to a mobile-based platform targeting medical diagnosis; this was SignSupport version 2. Mutemwa s SignSupport made use of pre-recorded SASL videos inside a mobile browser. The prototype ran on a Symbian mobile phone and employed a guided set of web pages with a combination of SASL videos and English text to enable a Deaf person to tell a doctor how s/he is feeling and to pin-point where it hurts. At each step, the Deaf user responds to a series of questions presented in SASL, and finally enabling the system to convey how the Deaf user is feeling into simple English for a doctor to understand [14]. SignSupport was implemented to be stored both on and off the mobile phone [20]. SignSupport ran on a Symbian browser from XHTML pages embedded with SASL videos, without the use of a third party media [21]. Most of the SASL videos were stored on the phone. Implementing SignSupport this way meant that no network structure was required and Deaf people did not have to pay for data downloads when using SignSupport. The system was developed in a context free-manner which made it possible to add and update videos on the phone easily. Mutemwa s system was a medical system but not an expert system, and did not use artificial intelligence and it did not give a diagnosis. Rather, once the Deaf person has completed answering the questions shown in sign language, the mobile mock-up produced English text representing what the Deaf person has defined, and can be handed to the doctor to read what the problem is in plain English [21]. Our system then learns from Mutemwa s work and moves the scenario from the doctor to the pharmacist. SignSupport therefore is in some way or the other influenced by the work that has been done in the past number of your, this means we now have a clear understanding of what Deaf users expect from mobile communication devices. 3

6 The Aim(s) of the Research The system aims to establish precise communication between a Deaf person and a pharmacist in a hospital pharmacy. The system uses text to communicate with a pharmacist and then the pharmacist makes selections that correspond to SASL videos stored on the phone. The videos contain a real signing person and not an avatar. These SASL videos help instruct the patient on how to use the medication. The main objective in developing this prototype is to determine how useful it would be to a Deaf patient whose first language is SASL, and to discover what areas of the system require research and development to make it more effective as a communication tool in this limited domain exchange between a pharmacist and a Deaf patient. We aim to improve the communication flow in order to discover better methods of organizing, structuring and presenting information to all parties involved; to find an interface that both users can accept and understand. Rationale Deaf people mainly use signed language to communicate. This group of people is often functionally illiterate [30][2], causing miscommunication and misunderstanding with hearing people that cannot sign, e.g. a pharmacist. When a Deaf patient collects medicine at a public hospital pharmacy, failure to understand the medication instructions is largely due to language barriers. In addition, Deaf people can have misconceptions about medicine intake, for instance believing that all medicines have to be taken after meals. This can also lead to a consequence of misuse of the medication. Furthermore, Deaf patients should have privacy when communicating with a pharmacist, potentially without the use of an interpreter. During the consultation, Deaf patients require to understand the concept of their medicine therapy and its instructions in detail [14]. When a Deaf person wishes to communicate with a hearing person in a public setting there has to be an interpreter present unless the hearing person can sign. In general, public workers in South Africa cannot sign. Currently technology in South Africa is not as supportive of the situation as in other places such as England where public workers can use TESSA to communicate with Deaf people [7]. In South Africa, Deaf people still rely on their family members and neighbors to help in such situations. A mobile phone is a common communication tool for Deaf people, thus they are familiar with a mobile phone even though they do not use its voice capabilities. Ranges of cheap smart phones are now available and 4

7 popular among Deaf people, and we can avoid data charges by storing the video on the phone. The pharmacist can interact with the application s interface by tapping information, selecting from provided options and taking a picture of each medicine to deliver and transform the complete medication instructions in a signed language video format for the Deaf patient. In this way the Deaf patient can understand in SASL the concept of the medication, each medicines purpose, how to take it, when to take it, and the pharmacist s recommendations/warnings. In addition, the Deaf patient can easily recall information on medication(s) from the photo(s) taken by the pharmacist. The cycle of each medicine explanation was designed by following the strategy of Watermeyer and Penn [36] In addition, if the patient has any question concerning a medicine, the patient can also provide information to the pharmacist via the use of the application. In the worst case scenario, when SignSupport is not able to clarify something, the pharmacist and Deaf patient can communicate with each other in real time via an interpreter with Skype to clarify communication concerning the medication(s). Patients will need to have a high-end smartphone; most of these phones cost less than R4000 [24]. 3G-Video Call Rates Peak Standard Off-peak MTN to MTN R3.00 R3.00 R3.00 MTN to VodaCom R3.00 R3.00 R3.00 Table 1: Showing 3G-Video Call Rates for MTN service provider 3G-Video Call Rates Peak Standard Off-peak VodaCom to VodaCom R1.40 R1.40 R1.40 VodaCom to MTN R1.40 R1.40 R1.40 Table 2: Showing 3G-Video Call Rates for Vodacom service provider Unfortunately, this will incur data charges as shown in Table 1 and 2, and also require a phone with a front-facing camera; both expensive, yet optional [24][19][34]. Table 3 shows the price ranges of a 3G video call from both Vodacom and MTN for duration of 5 minutes. 5

8 Service Provider (3G video calling) Cost per 5min (ZAR) MTN to MTN and others R15.00 VodaCom to VodaCom and others R7.00 Table 3: Showing 3G-Video Call Rates for Vodacom and MTN per 5 minutes Table 3 shows that for every 5 minutes the user spends making a 3G video call they could be spending lost of money. Although this seems like a small amount over time the cost accumulates and the user ends up not being able to maintain this kind of expense, and for this reason we decide to use canned SASL videos to minimize service costs as much as possible. Literature Survey Introduction This section introduces work related to our project. Work related to communication applications for medical use and everyday use for Deaf people is presented. Technology details are also mentioned in this chapter. The first section is about medical expert system and how it relates to our solution. The relay systems section and how the communication links the applications to Deaf and hearing people. Translations section discusses to convert sign language into text/video and back using AI techniques, this is where our implementation can learn some tricks relating to limiting the communication domain to tighten out conversation structure. Mobile Sign language communication is discussed because is closely related to our mobile system here we look at the key area relating to video play on a mobile phone. The last section concentrates on the challenges of developing a communication system in a limited domain and for Deaf users. Over the years Deaf people have created and used sign among themselves. The problem comes when a Deaf person wants to communicate with a hearing person, usually both will be frustrated shortly before giving up. For this reason there has been attempts to design smart devices that can work as automatic interpreters between a Deaf person and hearing person. These attempts have now moved onto the mobile platform because mobile phones help improve positive factors such as mobility and liability, which would make sign language more attractive and valuable. It is possible to use XML on a mobile device to render a user interface, which will make it easy for Deaf and hearing users to communicate. Deaf people mostly make use of SASL to 6

9 communicate with each other. There are two groups of Deaf people who use SASL; the first being Deaf those who used to be able to hear and speak at first but then became hard of hearing for some reason and now have to communicate in sign language but can read and write; and the second are those being Deaf who were born unable hear or speak. Both groups may have a low level of basic education, but the second group has worse numbers of illiteracy [30][2]. Deaf people have difficulty conversing with non-signers, watching movies and television, and reading books and the World Wide Web (WWW). In communicating it is essential to get straight to the point, but this can be a difficult thing to achieve if the language used for the conversation is not the preferred language for either one or both people involved. When both people use the same language that they are comfortable with, then the conversation will not only become easier to express each other s ideas but more ground can be covered in terms of information shared on a specific subject [16]. Relay systems Telecommunications Relay Service (TRS) is an operator service that allows people who are Deaf to place calls to standard telephone users via a keyboard or assistive device. Originally, relay services were designed to be connected through a TTY or other assistive telephone device. Services have gradually expanded to include almost any real-time text capable technology such as a personal computer, laptop, mobile and phone. The most common type of TRS call, involves a call from a person who is Deaf and utilizes a TTY to a person who is Hearing and does hear and speak. In this call type, typed messages are relayed as voice messages by a TRS operator and vice-versa. This allows callers unable to use a regular telephone, to be able to place telephone calls to people who use a regular telephone, and vice-versa. When the person who is Hearing is ready for a response, it is customary to say Go Ahead or GA to indicate that it is the TTY user s turn to talk and Stop Keying, SK, or Ready to hang up when hanging up, and vice-versa. Most Deaf people make use of text communications through Short Message Service (SMS), and fax to communicate within the Deaf communities and with hearing people. SMS is the most used and has since replaced the once popular tele-typewriter (TTY) because most mobile phones have this service available on their mobile phones. s and faxes are services only considered usable at work places and not for social reasons. Deaf people preferred not to use TTY systems to communicate with a hearing users for social or business purposes, only a few of the user had a TTY system. In 7

10 survey that was placed online in Germany from April 2005 to March 2006 and a total of 102 Deaf people responded to the online survey. Of the Deaf people who took the online survey, 55% were male, about 50% were Deaf from birth while the rest had varying degrees of being deaf, all ages were covered with the majority of participants being 27% from years old, 44% used German Sign Language as their first language, 69% said they were educated mostly in Deaf schools, and 49% said they had finished high schools education. The survey results were as follows: most Deaf people said they preferred not to use wireless application protocol (WAP). Only a few said they used WAP, and they visit sites that had weather forecasts, Deaf News (from Deaf politics, Deaf sports, Deaf events, Deaf information), course information, games and picture download. Most of the Deaf users said there Keypads were OK, but a few said they preferred a QWERTY layout keypad, while others wanted a bigger keypad and most preferred Nokia phones. Almost a majority of the Deaf participants said they paid their own SMS bills. About a third of the Deaf participants said they responded to SMSs immediately while others said from one hour or the same day. Most of the deaf people said they have used a computer before, and they used it mainly for s, World Wide Web, chat applications, and sending SMSs [22]. A group of computer scientists and linguists are working on a portable communication aid for Deaf-blind people. A special surface for blind people has been developed that converts the sign language into text then to brail and back; the surface is connected to any electrical screen to show the resulting sign language. The portable communication aid is meant to allow the deaf-blind people to communicate with others without the help of an assistant. The system is composed of two major units-one for the deaf-blind persons and the other for sighted persons. The deaf-blind person can send messages by typing on a Braille terminal and the messages will be converted to be Mandarin phonetic symbols that are then displayed on an LCD display to be read by a sighted partner. Then the sighted partner can send messages back by typing on a simple keyboard and the messages will be displayed on a Braille display to be read by the blind-deaf person. The aid has been designed to be effective, reliable, and inexpensive [27]. Translation systems Translation systems attempt to convert sign language into text/video and back using Artificial Intelligence (AI) techniques. Most of the current sign language recognition systems use specialized hardware and are very person-dependent. Furthermore, most 8

11 approaches focus on the recognition of isolated signs only, or on the simpler case of isolated gesture recognition which often can be characterized just by their movement direction [41]. An experimental system called TESSA was developed that aimed to aid transactions between a deaf person and a clerk in a Post Office by translating the clerks speech to sign language [7]. A speech recognizer recognizes speech from clerk and the system then synthesizes the appropriate sequence of signs in British Sign language (BSL) using a specially developed avatar. By using a phrase lookup approach to language translation, which is appropriate for the highly limited domain in a Post Office, TESSA is an interactive system that operates in a very restricted domain by inly focussing on the important phrases that occur in a Post Office transaction. They closely developed TESSA for the post office because most of the conversations are predictable and simple to follow. The TESSA system does not use a real interpreter to do sign language. They decided to use an avatar for their system. This is because they have been developing an experimental system that uses a virtual human (avatar) to sign television subtitles [7]. Six pre-lingually profoundly deaf people whose first language is BSL and three post office clerks took part in the evaluations of the system. The evaluations took place over three sets of two days. Two deaf people and one clerk attended for each pair of days. The first day started with completion of the first part of a questionnaire. Each deaf participant then alternated between identifying a block of signed phrases and attempting a block of staged transactions. The results of the study went as follows: Some deaf participants who said that communication in the post office usually upset or worried them and some thought using TESSA in the Post Office would not bother them at all [7]. Millions of dollars were invested in developing TESSA and lots of groundbreaking research answers came out but TESSA became a failure and was never really popular with deaf people. Lisa Watch a media relations officer at the Royal National Institute for Deaf People said Obviously there are some reservations within the deaf community. They are concerned that they will end up being forced to communicate with computers the entire time, when what they would prefer is more human sign language interpreters. Stephen Cox, professor of Britain s University of East Anglia, wrote in an interview there are several thousand post offices in the U.K., and TESSA is not currently integrated with the systems used in them, so it requires an extra PC to be purchased and installed, plus two screens, also the post office staff need to be trained. It really isn t worth it unless there are a significant number of deaf users at that Post Office [39]. 9

12 The goal of ViSiCAST was to improve the quality of life of Europe s deaf citizens by widening their access to services and facilities enjoyed by the community at large. The project identified a number of aspects of life where the integration of deaf individuals in society would be improved if sign language communication was available: such as access to public services, commercial transactions and entertainment, learning and leisure opportunities including broadcast and interactive television and the e-commerce and the World Wide Web [8]. Any aspect of society they identified as an area of study was then considered closed of and limited. Only the most frequently occurring activities would be part of the final result. In the field of limited domain for Deaf people ViSiCAST followed were TESSA left off. ViSiCAST was a major new project funded by the European Union, aiming to provide improved access to services and facilities for deaf citizens through sign language presented by a virtual human, or avatar. Just as TESSA, ViSiCAST uses a virtual human and instead of capturing sign language directly from video, which is more desirable as reported [25][40][26][4][17], they claim such approaches are not yet practical. So their alternative was to capture the signs using individual sensors for the hands, body and face controlled by a separate thread (event loop) that tracks the skeleton. The ViSiCAST team started their first prototype with real sign language videos and later changed to using an avatar. A major application area for ViSiCAST is to bring virtual human signing into the World Wide Web environment, through the development of a browser plugin, which incorporates the avatar software. This will provide a basis for the provision of signing services for Deaf users of the World Wide Web [8]. By the end of the project the ViSiCAST project was able to achieve a number of goals: A face to face transaction virtual signing system was tested in UK Post Offices by 2 clerks and a panel of 5 deaf users [31]; The signing preparation and virtual signing system was used to prepare and present at least 4 sample television programmes with virtual signing, and the feasibility of broadcast transmission of virtual signing within MPEG-2 multiplexes was established for both transmissions in the UK and Germany [31]; The World Wide Web tools were installed and used in Web sites, and these were evaluated by representative members of the deaf community. The aim was to install examples of the annotated Generative Modeling Language (GML) based signing on at least 5 of the Web sites of the ViSiCAST participants [31]. To achieve these objectives the project was structured to have three applicationorientated Work Packages, each focusing on the technical issues in delivery for that 10

13 specific application area, and two enabling technology Work Packages, focusing on virtual signing, sign language representation, and sign language synthesis from conventional textual sources. The TESSA system will be enhanced in a number of ways, for use in other contexts, such as health centres and hospitals, advice services, and shops. The system aims to relax the communication constraints between the service provider and a Deaf person [31]. Mobile Sign language communication Most of the technologies discussed thus far are implemented on PC s. MobileASL is a video compression project that seeks to enable mobile wireless cell phone communication through sign language [5]. MobileASL works on commercial phones that are accessible to Deaf people. The motivation to MobileASL is to make as clear a sign language video as possible to transmit over the network. They conducted user studies with members of the Deaf Community to determine the intelligibility effects of video compression techniques that exploit the visual nature of sign language. Unfortunately, the Deaf Community in the United States. cannot yet take advantage of this new technology. Preliminary studies have strongly suggest that even today s best video encoders cannot produce the quality video needed for intelligible American Sign Language (ASL) in real time, given the bandwidth and computational constraints of even the best video cell phones. MobileASL concentrates on three major areas when manipulating video for sign language use. These areas are as follows; Bit rates- three different bit rates were studied: 15, 20, and 25 kilobits per second (kbps). These values were chosen in an attempt to accurately portray the current United States mobile phone network: the optimal download rate has been estimated at 30 kbps whereas the upload rate is considerably less, perhaps as low as 15 kbps [5]. Frame rates- two different frame rates were studied: 10 and 15 frames per second (fps). Preliminary tests with a certified sign language interpreter revealed that 10 fps and 15 fps were both acceptable for intelligible ASL. The difference between 30 fps and 15 fps was negligible whereas at 5 fps signs became difficult to watch and fingerspelling became nearly impossible to understand. Frame rates of 10 and 15 fps were chosen for this study to investigate the tradeoff of fewer frames at slightly better quality or more frames at slightly worse quality for any given bit rate [5]. 11

14 Region of interest- three different region-of-interest (ROI) values were studied: -0, -6, -12, where the negative value represents the reduced quantizer step size, out of 52 possible step sizes, in a fixed 6Å-10Åmacroblock region around the face (a single 320Å-240Åpixel frame is composed of 15Å-20Åmacroblocks). Reducing the quantizer step size in this region results in less compression (better quality) in the face region and more compression (sacrificing quality) in all other regions for a given bitrate [5]. Medical expert systems A medical expert system is software that is programmed using A.I. A medical expert system uses huge libraries to phrase questions and based on the answers from the questions, calculates the likelihood of a disease [32]. Expert systems have to evaluate all possible diseases listed in the database from the user s response to the phrased questions. Well defined medical expert systems contain two knowledge types: firstly, the objective knowledge which can be found in textbooks, and secondly, subjective knowledge which changes frequently and is limited. The main criteria for the determination of the success or failure of a medical expert system are the accuracy and the rate of the diagnosis [20]. The number of diseases that can be diagnosed by the system should also be high enough so that it can be distinguished between the common symptoms. A medical expert system can be used from the patient s sideto diagnose the disease s/he is suffering from. One example of a medical system is WebMD s symptom checker is an online advanced Decision Support Systems (DSS) used to provide information while awaiting evaluation with a doctor or provide a user with additional information after consulting with a doctor [13][38]. Symptom checkers do not replace face-to-face communication but they provide additional diagnosis on the most common diseases. The diagnosis is not exhaustive and there can exist many diagnoses that the system missed. The system interface begins by showing a full image of a human body. A user can then click on the body part they wish to know more about. The system will then ask questions related to the clicked body part, which are presented one at a time in simple English. Medical expert systems are not directed towards Deaf people because they do not have a Deaf user interface. 12

15 Challenges There are many factors hindering the developments of assistance devices. First, in many countries around the world, there is very limited funding to support the development and manufacture of such devices and technological advances. Second, the Deaf population is very low and scattered. This represents a limited market and prohibitive cost [27]. As a result most Deaf people are moving towards internetbased technologies, SMSs, and Instant Messaging applications for communication [22]. These communication solutions are very expensive in the South African context and are thus not an effective medium of communication for Deaf users since most Deaf people in South Africa are poor. The majority of the communications on the mobile phone networks come from hearing people, which means mobile phones do not fully support technologies that allow Deaf users to communicate on the mobile phones. For deaf people as well as everyone else, the ability to communicate while on the move is now seen as essential to business, commerce, individual lifestyles and everyday social. Among Deaf people, the rate of use is estimated to be much higher than among hearing people [5]. However, one indicator is that there is an increase of mobile phone ownership among Deaf people, most of whom are unable to use them for voice messages. Research has shown that Deaf people readily adopt texting via SMS [20]. A survey published by the National Association of the Deaf showed that sending s by mobile phones, personal computer or digital pager has become so popular that for some, it has replaced the once widely used TTY [21]. Recently IM has become wireless from its traditional desktop applications to mobile phones. SMS allows Deaf people in Australia and some other countries private access to business and social information on their mobile phones. It is predicted that SMS will increase bonds between communities and creating understanding with those not physically present and soon after that video calling will be even more accessible to Deaf people as costs decrease [27]. Research Problem The research problem is to devise a solution on a mobile phone in a limited domain exchange environment that can improve communication between Deaf patients and pharmacists. This will reduce problems of non-compliance to treatment protocol due to miscommunication. 13

16 Research Methods The initial SignSupport mock-up on a personal computer showed potential to help Deaf users use a mobile phone to communicate with people that do not sign [14]. SignSupport version 2 ran on a mobile phone browser and allowed for multiple content modules, as the Deaf person visiting to a doctor is just one of them. SignSupport version 3 is for pharmacist and will run on a mobile phone with SASL videos stored on the phones memory card and thus will look to solve problems related to limited domain communication exchange. We define the limited domain as the interaction that takes place between the Deaf patient and the pharmacist in a public pharmacy. This allows us to concentrate on the conversations that take place between patients and pharmacists. To extract these conversations we had to conduct role plays and sift through the most commonly use phrases by the pharmacist. These phrases are central to the video that will be later stored on the cellphone to explain medicine intake to the patient. To cater for videos that are not stored on the phone and for unpredictable situations the application will also allow for breakout. When there is a communication breakdown between the pharmacist and the Deaf patient, the option of using video relay will be integrated into the system. This situation will be considered outside our domain. To design the system for both users we break it into two different sections, one for the pharmacist and the other for the Deaf patient. The side of the pharmacist is more text-based and all input-output is text. This is because pharmacists can read and write. The side of the Deaf patient is then video and picture based because most Deaf people can not read and write English [2]. To eliminate poor communication and confusions between a Deaf patient and a pharmacist a solution based on a mobile phone was co-designed by Deaf people in a local community, pharmacists and us [6]. For the re-evaluation of the system we will continuously revisit the two paragraphs above in order to refine the and restrict out domain to be more feasible and effective as possible. Our approach is to always use iteration design so we can always improve our solution based on the feedback we get from all stakeholders involved. Delimitation of the Study The assumption for this study is that sign language communications between a Deaf person and a pharmacist is possible on a mobile phone in South Africa at an affordable price as already mentioned above. This kind of communication can reduce the rate of 14

17 Questions Answer a Answer b Are you Male/Female Male Female Do you smoke? Yes No Do you have allergies? Yes No Are you on any meds? Yes No Do you consume alcohol? Yes No Table 4: Background questions from patient data in SASL video form on the sd-card Categories Gender Smoker Allergies On medication Consume alcohol User answers Male Yes No Yes No Table 5: Patient background answers from patient seen in text by pharmacist wrongful medicine intake and breakdown some medication misconceptions found in the Deaf community. There will only be so much dialogue collected between the Deaf patient and the pharmacist that the system can handle when it comes to matching the pharmacist s instructions with the sign language video. We have already collected data in a form of role-play that allows us to predict what questions and answers will emerge when a pharmacist interacts with a patient and from these interactions we will structure the conversation flow such that the pharmacist will have to ask the least amount of questions which lead to the patient answering even less questions. Figure 4 shows the kind of questions the patient will have to answer as part of their background information all these questions will be represented in sign language video already stored on the phone and translated to English text for the pharmacist to read. This creates a scenario where the pharmacist does not have to ask the patient questions like, do you have any medicine allergies? Or do you have access to clean water? The idea is that the patient will answer questions from Table 4 in SASL video format and the pharmacist will see the answers presented as in Table 5. A great limitation is that since our prototype will runs on a mobile phone and by their nature mobile phones have very limited resources, more specifically space limitations and processing power will have to be carefully managed. SignSupport version 3 will not replace the pharmacist, but rather assist in archiving the goal of communicating and dispensing medicine to Deaf patients in public hospital pharmacies. 15

18 Implementation and Methodology The aim for this project is to study and improve communication in a limited domain between Deaf patients and hearing pharmacists who cant sign; this will help prevent problems of non-compliance to medical treatment in South African hospital pharmacies. The procedure for the methodology will begin by discussing the participants that will involved in the study, the material that will used during the experiment study, the design of the testing process and lastly the procedure that will be used to carry out the testing [12]. Participants The first group for this study is Deaf people who only use sign language as their primary source of communication with other deaf people and hearing people who do not sign. This group of participants is not required to be able to read or understand a written language, but the ability to use a smartphone and SASL to communicate is mandatory. From such a large group this study will only focus on DCCT staff members who were involved in the previous study done by Chininthorn. The reason for this is that they familiar with the SignSupport systems and they were part of the initial design of the system during Looijesteijn s, Mutemwa s and Chininthorn s study. The second group is composed of pharmacists from UWC School of Pharmacy and since this prototype will have a section that a pharmacist will interact with. The work done by Chininthorn clearly states that the design of the system requires the input of pharmacists [6]. Materials The primary experimental tool for this project is the prototype. The experiment will be on a mobile phone, using the sign language videos stored on the phones memory card and the dialogs from pharmacist to control and synchronize the resulting SASL video. For the user trial a storyboard will be used to inform the people participating the user trial what is required of them and what they are supposed to do and how to behave in the role-play. During the user trials a video camera will be used to record the sentences extracted during role plays to form part of the SASl videos on the phone. This study composes of a system that has a sign language video, which will be recorded, with a person who can sign in SASL. After the videos are recorded they will be added to the system using a desktop computer. Semi-structured interview and 16

19 questionnaires will be used to gather information from the user for the user trials. Procedure To collect the conversations that are most prevalent in the hospital pharmacy a role play was conducted that imitates a medication dispensing situation which takes place in public hospital pharmacy. Each role play composes of a pharmacy student participant and a simulated patient actor. Each participant will read the information sheet about the details of this study and sign consent from before entering the role play room with camera installed in it. The camera is connected to a TV screen which a pharmacy researcher who is part of the research will be observing the interaction and the dialogs during the medication dispensing process. Each participant was ask to participate in two rounds of the play. The first round he/she will dispense prescribed medicines for treatment acute disease. The second round he/she dispense prescribed medicines for treatment chronic disease. The brief of each play will be described to the participant before starting the play per round. The role plays are the start of developing a system in the limited domain so we followed the Watermeyer and Penn [36] medication explanation strategy which is meant to promote the patients understanding of the provided medication instructions. All SASL videos will be pre-recorded based on the study of actual communication patterns during the testing. This is the beginning of backend development of the system. The front-end of the application involved similar strategies. The design for of the system emerged from a process we call community based co-design which was implemented by Chininthorn. She employed conventional user-centred methods; however, she did not consider the Deaf participants and pharmacists as experimental subjects but rather as experts in their communication realities. This approach proved to be very promising to effect real social change and give birth to a user interface that is acceptable for both Deaf users and pharmacists. As can be seen from Figure 1 (iterative model) all the methods and strategies discussed above and in this project in general will be repeated over and over again in oder to refine and make the software system as effective as possible. This will ensure that the system is as close to the desired/expected result as possible. Re-evaluating the system in this way will afford the opportunity to gain user acceptance. 17

20 Requirements gathering Requirements analysis Waterfall model Implementation Verification User trial Prototype testing Coding Literature survey User survey Iterative model Prototype design Figure 1: This is the project design plan that will be followed; shown here are two software engineering methods, the iterative cycle preceded by the waterfall model. Design strategies In this study two kinds of software engineering methods will be used at different stages of development, the first is the waterfall method, used to start off the process of requirements gathering and analysis the second is the iterative development software design principle will be used as shown in a cycle of elements starting with the literature survey and ending with the report of the results compilation, see Figure 1. Procedure (waterfall development) Methodology steps see Figure 1. Requirements gathering and analysis - this is done by performing role-plays with student pharmacists in order to collect the sentences used during medicine dispensing. This process also studies the behavior of both the patient and the pharmacist. Then when all the data has been collected we will analyze and record SASL videos from the resulting sentences The design follows the requirement analysis development, and this is to design the system, from the programmers point of view. This design involves the 18

21 system architecture and eventually how the whole prototype will behave. An implementation will be performed after the final design stage [1]. The prototype will be developed quickly; this will most likely be the throw awayprototype. In this phase we really want to see that the system can indeed be build given the user requirements. Verification, after the implementation stage, this is where we ask Deaf people and pharmacists if we are building the right prototype? And are we building it correctly. An implementation will be performed after the final design stage [6]. The prototype will then be tested and evaluated by the intended users (Deaf patients and pharmacists); some experiments will also be done on the prototype, to evaluate the intelligibility of sign language video on the prototype. Procedure (iterative development) Methodology steps see Figure 1. The project will begin with a literature survey, which will determine what can be done and what cannot be done but can be recommended for future work. This will start by looking at the previous study done on the concept [14]. This is followed by a user survey, which will determine what the intended users of the prototype will expect of the system. A requirements analysis development is performed from the user survey to determine which of the requirements gathered can be implemented and how long it might take. The design follows the requirement analysis development, and this is to design the system, from the user interface to the way it operates. An implementation will be performed after the final design stage [1]. The prototype will then be tested and evaluated by the intended users (Deaf patients and pharmacists); some experiments will also be done on the prototype, to evaluate the intelligibility of sign language video on the prototype. A report will be compiled to give feedback of the overall results from the testing, evaluation and experiments. 19

22 Data Collection Both qualitative data and quantitative data will be analyzed. The qualitative methods that will be used are semi-structured interviews, video camera recordings and focus group. During the user requirements data will be collected using from the questionnaires the Deaf users will fill out, and the response from the semi-structure interview. During the user trial data will be collected using a video camera to capture the user trail, questionnaires filled in, and the response from the focus group. The prototype will be tested form two ends: the first is the pharmacist side, where a pharmacist fills in a prescription using the prototype following onscreen instructions and the second end is from the Deaf patients side where they check if check to see if all the medical instructions are entered and the corresponding sign language videos on the phone make sense. 20

23 Thesis Chapter Outline 1. Introduction 2. Related work 3. Methods 4. Research findings and analysis 5. Conclusion Dissemination of Research Once this study is completed a working prototype would have been developed that can be installed mobile phones and can be used for communicating with pharmacist at any public hospital pharmacy. All the prototype and the source code will be available on the BANG website and on my masters page and members of DCCT will be encouraged to download the software together with source code and documentation. The thesis write up will be free to download as it is academic work and the University of the Western Cape will hold all the rights. Most of the research answers and information obtained in this study will be used in presentations, SATNAC papers and in the thesis, which will be handed in to the University of the Western Cape. 21

24 Glossary Computer-based mock-up - the SignSupport prototype designed by Looijesteijn an industrial engineer that used Adobe Flash but can only run on the computer [14]. Deaf - a person who primarily uses sign language as a means of communication, even though they are textually literate. DCCT - Deaf Community of Cape Town, a non-governmental organization, which owns the Bastian where user trials will be carried out [30]. Focus group - A group of people who have represented the target group on the computer-based mock-up design and testing, and who will preferable be used in the next set of trials based on the mobile-based mock-up. FPS - frames per second, the measurement of the frequency at which an imaging device produces unique consecutive images called frames [33]. IM - Instant Messaging, allows two people to chat electronically in real time [10]. Lookup dictionary - finding one or more videos from a set of videos items that have a specified property. Mobile-based mock-up - the SignSupport prototype designed by Looijesteijn an industrial engineer that used Adobe Flash that is supposed to run on the mobile phone browser [14]. Mobile phone - a long-range electronic device used for mobile voice or data communication over a network of specification. MPEG - a set of standards adopted by the moving pictures experts group for the compression of digital video and audio data or a file of data compressed according to those standards [37]. Semi-Synchronous - A type of two-way communication with little time delay, but allowing participants to respond in almost real time. SignSupport - a system that is supposed to ease communication between a Deaf user and a hearing user, in a semi-synchronous communication. SMS - Short Messaging Services, used to send text messages on mobile phones [18]. 22