MobileASL: Making Cell Phones Accessible to the Deaf Community

Size: px

Start display at page:

Download "MobileASL: Making Cell Phones Accessible to the Deaf Community"

Tracey Simmons
5 years ago
Views:

American Sign Language (ASL) MobileASL: Making Cell Phones Accessible to the Deaf

about 500,000-1,000,000 Deaf people in the U.S and most of Canada.

location, orientation, shape of hands and arms + facial expressions Usually uses hands,

Benefits: Current Technology for Deaf People (video phones) Set-top boxes Benefits:

(PDAs) Problems: Web cams ASL, not English Problems: English, not ASL 3 Requires high

U.S. cell phone network Challenges: Limited network bandwidth Limited processing power

1 American Sign Language (ASL) MobileASL: Making Cell Phones Accessible to the Deaf Community Richard Ladner University of Washington ASL is the preferred language for about 500,000-1,000,000 Deaf people in the U.S and most of Canada. ASL is not a code for English Signs usually occur within the sign-box Composed of location, orientation, shape of hands and arms + facial expressions Usually uses hands, but one-handed signing not uncommon Current Technology for Deaf People (text) TTY Benefits: Current Technology for Deaf People (video phones) Set-top boxes Benefits: Sidekicks and Blackberries (text, pictures, non-real-time video) Low bandwidth Mobile (PDAs) Problems: Web cams ASL, not English Problems: English, not ASL 3 Requires high bandwidth Not mobile 4 Our goal: ASL communication using video cell phones over current U.S. cell phone network Challenges: Limited network bandwidth Limited processing power on cell phones Architecture Cell phone user interface Sender Receiver Camera Player Encoder Decoder Transmitter Receiver 5 Cell Phone Network 6 1

at much lower Hz then PCs New mobile broadband services Higher bandwidth for download, not upload.

2 Cell Phone Network Constraints What about 3G? MobileASL is about fair access to the current network As soon as possible, no special accommodations Low bit rate constraint GPRS - Ranges from 30kbps to 80kbps (download) Low Power Cell phones run at much lower Hz then PCs New mobile broadband services Higher bandwidth for download, not upload. 7 8 Portrait Special Codec from Microsoft Asia Low Bandwidth, Low Power, small size video (160 x 10) May not be suitable for sign language Keman Yu, Jiangbo Lv, Jiang Li and Shipeng Li, 003 Codec Used: x64* Open source implementation of H.64 standard Doubles compression ratio over MPEG x64 offers faster encoding Main profile Off-the-shelf H.64 decoder can be used *The code is written from scratch by Laurent Aimar, Loren Merritt, Eric Petis, Min Chen, Justin Clay, Mans Rullgard, Radek Czyz, Christian Heine, Alex Izvorski, and Alex Wright. It is released under the terms of the GPL license User Studies MobileASL Focus Group 4 Deaf people, mid-0s to mid-40s, Open ended questions: Physical Setup Camera, distance, Features Compatibility, text, Privacy Concerns ASL is a visual language Scenarios Lighting, driving, relay services, 11 1

Implications of Focus Group I don t foresee any limitations. I would use the phone anywhere: the grocery store, the bus, the car, a restaurant, anywhere!

while eye movements were tracked Important regions of the video could be encoded differently * Muir et al. (005) and Agrafiotis et al.

Detailed grammar in face requires foveal vision Hands and arms can be viewed in peripheral vision * Muir et al. (005) and Agrafiotis et al.

3 Implications of Focus Group I don t foresee any limitations. I would use the phone anywhere: the grocery store, the bus, the car, a restaurant, anywhere! There is a need within the Deaf Community for mobile ASL conversations Existing video phone technology (with minor modifications) would be usable Eyetracking Studies Participants watched ASL videos while eye movements were tracked Important regions of the video could be encoded differently * Muir et al. (005) and Agrafiotis et al. (003) Eyetracking Results Mobile Video Phone Study 95% of eye movements within degrees visual angle of the signer s face (demo) Implications: Face region of video is most visually important Detailed grammar in face requires foveal vision Hands and arms can be viewed in peripheral vision * Muir et al. (005) and Agrafiotis et al. (003) 15 3 Region-of-Interest (ROI) values Frame rates, frames per second (FPS) 3 different Bit rates 15 kbps, 0 kbps, 5 kbps 18 participants (7 women) 10 Deaf, 5 hearing, 3 CODA* All fluent in ASL * CODA = (Hearing) Child of a Deaf Adult 16 Example of ROI Varied quality in fixed-sized region around the face Examples of FPS Varied frame rate: 10 fps and 15 fps For a given bit rate: Fewer frames = more bits per frame (demo) x quality in face 4x quality in face 17 (demo) 18 3

Questionnaire User Preferences Results 5 Bit Rate Frame Rate Region of Interest Average Participant Response 4 3 1 15 kbps 0 kbps 5 kbps 10 fps 15 fps 0 roi 6 roi 1 roi 19 Type of Encoding 0

4 Questionnaire User Preferences Results 5 Bit Rate Frame Rate Region of Interest Average Participant Response kbps 0 kbps 5 kbps 10 fps 15 fps 0 roi 6 roi 1 roi 19 Type of Encoding 0 Implications of results A mid-range ROI was preferred Optimal tradeoff between clarity in face and distortion in rest of sign-box Lower frame rate preferred Optimal tradeoff between clarity of frames and number of frames per second Results independent of bit rate User studies 1 Rate, distortion and complexity optimization Parameter Settings Input parameters Raw video H.64 encoder Compressed video Input parameters Raw video H.64 encoder Distortion Encoding time Objective: Achieve best possible quality for least encoding time at a given bitrate 3 input parameters # of options # of reference frames 16 motion estimation 7 partition size 10 quantization method 3 Total = 16x7x10x3 = 3360 tests/video clip 4 4

5 Time Complexity Tradeoff 30 kbps 10 ASL videos GBFOS Approach Chou, Lookabaugh, Gray, 1989 Choose input parameter that minimizes the slope on the convex hull and repeat. Parameter settings are not independent. Basic Compute slopes once. Iterative Recompute slopes after each parameter is chosen. 5 6 PSNR vs. Average Encoding Time User studies 7 8 Encoding/Decoding on the Cell Phone Encoding performance for high/medium/low quality settings with and without code optimization Implemented a command-line version of x64 on a cell phone using Windows Mobile Edition 5.0. Required significant modifications to the Linux based x64 codec. frames/second (average) x x high settings QVGA med settings QVGA low settings QVGA lowest settings QVGA Unoptimized high settings QCIF Wireless MMX optimized med settings QCIF low settings QCIF lowest settings QCIF 30 5

6 Examples of Low Frame Rates Demo User studies 31 3 User Interface Design: Goals Usable, intuitive, easy to learn Inspired by Deaf users Utilize existing knowledge (VP, Webcam, Sorenson ) Design stages: Story boards Paper prototype testing Digital prototyping Basic Interface Split Screen with Text

Call Set-up User studies 37 38 Current Work Dynamic Region of Interest Dynamic Region-of-Interest Skin detection algorithms Objective Metrics For ASL

skin detection algorithms to drive region of interest.

7 Call Set-up User studies Current Work Dynamic Region of Interest Dynamic Region-of-Interest Skin detection algorithms Objective Metrics For ASL Understandability Activity Recognition Fingerspelling, signing, listening Building the System Transmission, Receiving, Playing Packet loss on GPRS 39 Use skin detection algorithms to drive region of interest. Fast skin detection algorithms exist Demo 40 Objective Metric Importance Face Hands Signing Box Weighted MSE based on where the pixels are Objective Intelligibility Metric Average Intelligibility Score Subjective Intelligibility vs. PSNR R = PSNR (db) 4 7

Objective Intelligibility Metric I = 10log 10 55 F MSEF + H MSE H where F = 0.6 andh = 0.4 Objective Intelligibility Metric Average Intelligibility Score Subjective Intelligibility vs.

8 Objective Intelligibility Metric I = 10log F MSEF + H MSE H where F = 0.6 andh = 0.4 Objective Intelligibility Metric Average Intelligibility Score Subjective Intelligibility vs. Objective Metric R = Objective Intelligibility 44 Activity Recognition Possible Solution : Finger spelling requires a higher bit rate and/or frame rate for intelligibility than signing We want to minimize encoding complexity when not signing. Goal: Recognize these three states: finger spelling, signing, not signing Perform recognition in real time 45 Use H.64 motion vectors as features Use probabilistic techniques to automatically recognize activity Hidden Markov Models Kalman filters or particle filters 46 Building the System in C#: Really easy to develop GUIs. Developers can only use their predefined interface for the camera. The interface is simple, but extremely limited. In C++: GUI development much more complex. Accessing camera requires knowledge of windows COM system. Thanks Co-PIs Eve Riskin and Sheila Hemami Graduate Students Anna Cavender, Rahul Vanam, Neva Cherniavsky, Frank Ciaramello, Dane Barney, Carl Hartung Undergraduate Students Jessica DeWitt, Loren Merritt, Sam Whittle National Science Foundation

MobileASL: Making Cell Phones Accessible to the Deaf Community. Richard Ladner University of Washington

MobileASL: Making Cell Phones Accessible to the Deaf Community Richard Ladner University of Washington American Sign Language (ASL) ASL is the preferred language for about 500,000-1,000,000 Deaf people