Can You See Me Now? Communication Over Cell Phones for Deaf People. Eve Riskin Richard Ladner Computer Engineering University of Washington

Transcription

1 Can You See Me Now? Communication Over Cell Phones for Deaf People Eve Riskin Richard Ladner Computer Engineering University of Washington

2 Co-PI Thanks Sheila Hemami (Cornell) Graduate Students Anna Cavender, Rahul Vanam, Neva Cherniavsky, Frank Ciaramello (Cornell), Dane Barney, Jaehong Chon Undergraduate Students Jessica DeWitt, Loren Merritt, Sam Whittle National Science Foundation, Intel, Microsoft 2

3 American Sign Language (ASL) ASL is the preferred language for about 1,000,000 Deaf people in the U.S and most of Canada. ASL is not a code for English Signs usually occur within the sign-box Composed of location, orientation, shape of hands and arms + facial expressions Usually uses 2 hands, but one-handed signing not uncommon 3

4 TTY Current Technology for Deaf People (text) Benefits: Sidekicks and Blackberries (text, pictures, non-real-time video) Low bandwidth Mobile (PDAs) Problems: English, not ASL 4

5 Current Technology for Deaf People (video phones) Set-top boxes Web cams Benefits: ASL, not English Problems: Requires high bandwidth Not mobile 5

6 Our goal: ASL communication using video cell phones over current U.S. cell phone network Limited network bandwidth Challenges: Limited processing power on cell phones 6

7 What about 3G? 7

8 Cell Phone Network Constraints MobileASL is about fair access to the current network As soon as possible, no special accommodations, no extra cost Low bit rate constraint Low Power Cell phones are much less powerful than PCs New mobile broadband services Higher bandwidth for download, not upload. 8

9 Encoder/Decoder Used: x264 Open source Good compression Fast encoding 9

10 Eyetracking Studies Participants watched ASL videos while eye movements were tracked Important regions of the video could be encoded differently * Muir et al. (2005) and Agrafiotis et al. (2003) 10

11 Eyetracking Results 95% of eye movements within 2 degrees visual angle of the signer s face (demo) Implications: Face region of video is most visually important * Muir et al. (2005) and Agrafiotis et al. (2003) 11

12 Mobile Video Phone Study 3 Region-of-Interest (ROI) values 18 participants (7 women) 10 Deaf, 5 hearing, 3 CODA* All fluent in ASL * CODA = (Hearing) Child of a Deaf Adult 12

13 Example of ROI Varied quality in fixed-sized region around the face 2x quality in face 4x quality in face 13

14 Problem Raw video Bit rate Encoder Parameters Real time? H.264 Encoder (x264) Fast encoding speed Compressed Video at given rate Best quality application Parameters are similar to knobs. As a knob is moved from lower to higher setting: quality improves while speed suffers H.264 Encoder 14

15 Selecting the Encoder Parameters Distortion data point Convex hull Exhaustive search: Run the encoder for all possible settings. Obtain vs. encode time plot Choose settings on the Distortion-Complexity convex hull Extremely time consuming! Complexity 15

16 Our Fast Algorithms Rahul took a MONTH to get data for last graph! We developed GBFOS-basic and GBFOSiterative algorithms to select good encoder parameter settings with fewer encodings We get similar results to exhaustive search for only encoding 1% and 8% of the videos 16

17 Selecting Encoder Parameters Both algorithms explained using an example: Four parameter setting variables: number of reference frames, partition size, sub pixel motion estimation and quantization method Other settings are constant 17

18 GBFOS-Basic Algorithm Obtain D-C plot for each parameter by setting other parameters to their best PSNR option Choose convex hull points and corresponding parameter settings and slopes Distortion-complexity plot for Number of Reference frames 18

19 GBFOS-Basic Algorithm Reference frames 16 Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4 Subme 7 Trellis 2 First parameter setting selected 19

20 GBFOS-Basic Algorithm Reference frames 16 Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4 Compare slopes Subme 7 Trellis 2 Find the parameter with the least slope 20

21 GBFOS-Basic Algorithm Reference frames 7 16 Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4 Subme 7 Trellis 2 Update the slope and setting of Ref. frames We now have the second parameter setting 21

22 GBFOS-Basic Algorithm Reference frames 7 Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4 Subme 4 7 Trellis 2 Compare the slopes again and find the parameter with least slope Update the slope and setting of Subme 22

23 GBFOS-Basic Algorithm Reference frames Partition sizes 7 P8x8 Subme 1 Trellis 1 The process is repeated to obtain the final parameter setting 23

24 GBFOS-Iterative More number of encodings Better performance Initialization and first iteration same as the GBFOS-Basic Algorithm 24

25 Results Maximum PSNR difference with respect to convex hull for: GBFOS-basic = db GBFOS-iterative = db 37.6 PSNR vs. encoding time for x264 encoder PSNR (db) Convex hull GBFOS-iterative GBFOS-basic Average Encoding time per frame (seconds) 25

26 Power consumption Phones less useful when battery drains quickly Expensive: processor cycles, transmission 26

27 Cycles per second Decode Encode fps 5 fps 1 fps Frame rate 27

28 Our Approach: Variable Frame Rate 28

29 Video processing Classify each frame as signing or listening Baseline method: simple differencing Works well in this case due to simple video Will not be robust enough for future use Machine learning method: support vector machines Features extracted, used to train/test Separates training data so classes are maximally apart 29

30 Features: H.264 information Type of macroblock Motion vectors 30

31 Features continued Features: (x,y) summary motion vector, face (x,y) summary motion vector, left hand (x,y) summary motion vector, right hand # of I blocks 31

32 SVM Classification Accuracy Test video Difference SVM Combine False neg gina1 88.1% 87.8% 89.9% 1.3% gina2 87.2% 85.2% 88.5% 8.0% gina3 88.8% 90.6% 91.1% 1.8% gina4 86.0% 86.6% 87.2% 7.1% Average 87.5% 87.6% 89.2% 4.6% 32

33 Future Directions More user studies Getting the (*%&#Q*!! system to run on a cell phone A field study with the phones with actual human beings MobileASL website: easl 33