Sign Language Recognition using Webcams

Transcription

1 Sign Language Recognition using Webcams

2 Overview Average person s typing speed Composing: ~19 words per minute Transcribing: ~33 words per minute Sign speaker Full sign language: ~200 words per minute Spelling out: estimate: 50 words per minute Up to 3x faster

3 Purpose and Scope Native signers can input faster Benefits: Hearing & speaking disabled Sign interpreters Just letters & numbers for now Additional complexity too much to handle Would require smaller distinctions

4 Research Related projects Using mechanical gloves, colored gloves Tracking body parts Neural network-based application Still images: 92% accuracy Motion: less than 50% accuracy Feature vector-based application Also about 90% accuracy on stills No motion tests

5 More Research Image techniques: Edge detection (Robert s Cross) Line detection (Hough transform) Line interpretation methods Chaining groups of lines Macro-scale templates Residual math Memory management

6 Testing Model Human interaction necessary General testing model: ~/syslab-tech $ \ >./main images/hand.png [DEBUG] Edge detect time: 29 ms Errors: 0 Warnings: 0

7 Program Architecture SERVER PROCESS Webcam capture Line detection IMAGE FEATURE OUTLINE LINE LIST FINGER POSITONS Edge detection Interpretation Attribute matching

8 Edge Detection Results Results: Outlines the important edges and not much besides Robert s Cross balances detection of major and minor lines Original image (800 x 703) Final image (800 x 703)

9 Cropping Results Remove useless rows & columns with no features Better contrast Original (800 x 703) Very large optimization Memory savings Area difference means order n 2 Result (633 x 645)

10 Finished! Line detection Recently finished tweaking sensitivities Still a few potential memory issues

11 Line Grouping Part of line detection Large optimization Iterate over an order of magnitude fewer items Easier to handle, more pronounced trends Examples of line groups, called chains

12 Line Interpretation Chaining groups of lines Templates Generation Template-based comparison Line residuals Use point coordinate averages Calculate average offset from average Easy to find height of finger

13 Sample Output After a typical run: 10 days, 6:12:19 until graduation!! bhood@testing ~/syslab-tech/src $./main hand.png Edge detection took 0.04 sec Image cropping took 0.00 sec Line detection took 0.17 sec (detected 1424 lines) Line chaining took 0.25 sec (detected 130 chains) Getting orientation took src (1 => ORIENTATION_FORWARD) Getting pinky pos. took 0.00 sec (2 => FINGER_BENT) Getting ring pos. took 0.00 sec (2 => FINGER_BENT) Getting middle pos. took 0.01 sec (2 => FINGER_BENT) Getting index pos. took sec (4 => FINGER_TUCKED) Overall process took 0.47 sec [TOTALCOUNT] allocated: , freed: ; leaked:

14 Timing Timing data from runs: To nearest hundredth of a second (0)Edge detection: 0.04 sec (1)Image cropping: 0.00 sec (2)Line detection: 0.17 sec (3)Line chaining: 0.25 sec (4)orientation: 0.08 sec (5)Pinky finger: 0.00 sec (6)Ring finger: 0.00 sec (7)Middle finger: 0.01 sec (8)Index finger: 0.00 sec (9)Overall process: 0.47 sec A little slow considering goal of real-time

15 The Mysterious Future Perfect line interpretation Work on memory management Am leaking large quantities (~50K) of memory Aggressive profiling needed Finish camera-computer interaction Device control must be precise, picky

16 The End! Code will be available to future years Contact me for a copy: byron@phareware.com