Microsoft Imagine Cup 2010 Thailand Software Design Round 1 Project Proposal Template PROJECT PROPOSAL DUE: 31 Jan 2010 To Submit to proposal: Register at www.imaginecup.com; select to compete in Software Design Challenge. Submit your proposal (accept both Thai or English) to email: icup_th@microsoft.com Application Name eyefeel Team Name SKeeK Team Members (Name & University/School) Member 1 Member 2 Member 3 Member 4 Full Name Krithee Sirisith Pichai Sodsai Thanasunn Phone Contact Email Postal Address Nonthawat Srichad Dilokpinitnun 0899842080 0816024416 0897994743 0892034411 Krithee_oak@hotmail.com 399/58 soi Pongpetnivet Prachachun Road Jaktujak Bangkok 10900 Thailand revolutionize_mix@ hotmail.com Room 4103. 222 moo 2 soi.ngamwongwhan 22 ngamwongwhan Road bangkhen nonthaburi 11001 Pae_kcy@hotmail.c om 5/1594 soi 13 Peachachun village Samakkee Road parkket nonthaburi 11120 thailnd nonthawat@hotmail. com 301/108 Tasai village soi 34 Tongsonghong Laksi Bangkok 10210 Institution Kasetsart University Kasetsart University Kasetsart University Kasetsart University Shirt Size L M L L Gender Male Male Male Male 1
Problem There are 364 millions of hearing-impaired people in the world, but only one deaf university (Gallaudet University in Washington, D.C.) can provides them with advanced education, which is far from enough. Every problem occurs from inability to hear or perceive sound and it makes them hard to communicate with other people. As you can see from present world, we and hearing-impaired people almost live in difference world. eyefeel is software project which aim to help hearing-impaired people to be able to communicate with normal people via augmented reality environment, showing conversation balloon and sign language animation of what people say. Finally, hearingimpaired can communicate with normal people. Furthermore, they can enter a normal school or university like normal people. Solution eyefeel provides way for hearing-impaired people to be able to communicate with normal people via augmented reality environment which combines speech recognition and face recognition system together in order to convert English speech from communicator to text and generate virtual conversation text balloon and sign language animation in real-time. Communicator simply speak to microphone in front of web camera, the system will recognize who is speaking and what is he/she speaking. eyefeel has additional features which are language translation, conversation logging and text to speech. User Scenarios 1. Steve is hard working man, he had a car accident and it made him deaf. His life has changed since then, he couldn't communicate with people like before. His life was in misery until he found out about eyefeel. With eyefeel he feels like he actually hears what people say, eyefeel generates virtual conversation text balloon next to communicator by speech recognition and face recognition system. He can hear people talk through his eyes and be able to work like normal. 2. Julie suffers from deafness since she was born. She had to go to private school for hearing-impaired people but she is fast learner and smart. Unfortunately, she couldn't study in famous university as her wish. But one day, she found out that Harvard university uses eyefeel which generates English text and sign language animation for hearingimpaired students and use it with video streaming in their lectures to store lecture data log and translate to another language, she didn't hesitate to apply to Harvard. With her intelligence and eyefeel, she graduated from Harvard and formed a great community to help people in the world. 3. Kenji family has moved to US but he never studied English before. He had trouble studying in US school, he even tried to use every translation tools to translate English speech to Japanese for him but he felt really odd with those tools and they were really hard to use, he almost gave up. One day his teacher suggested him to use eyefeel in the class. With augmented reality environment, he felt like his teacher actually speaks with Japanese language since it generates Japanese text next to his teacher face in real-time. He never had trouble studying in US since then. 2
4. Tim was born deaf, he never heard his father speaking before in his life. One day his father speaks with him via eyefeel and his father tells him how much he loves Tim. Tim can actually see his father speaking through his eye. It really touches his heart. Discuss Impact of Solution on Community/ Competitive Solutions eyefeel enables hearing impaired people to study in public school system and communicate with normal people. With language translation feature, everyone can use eyefeel to improve their learning capability, no more language gap in society. Unlike existed English speech to text and speech to sign language application, eyefeel provides augmented reality environment for user and makes communication feel natural since it integrated with face recognition system to track communicator face and generates text and sign language in conversation balloon next to the face in real-time. And conversation logging system makes conversation history tracking possible. And conversation and history conversation can translate to another language other than English. Discuss Business Viability of Application eyefeel has its own innovation and competitive solution with augmented reality environment which illustrate virtual conversation text and sign language balloon for users. Not only hearing impaired people who get benefit from this application, but everyone since it has multi-language translation and conversation logging system integrated with it. It can also be applied to another application like video streaming to provide real-time subtitle. And this application can be developed into portable device and hearing-impaired people can use it anywhere at any time. So with this approach it will have lots of support from community since eyefeel can help everyone especially hearing-impaired people. 3
Architectural Overview Figure 1: eyefeel system overview Technologies/Resources Used in the Application Core Technology - Microsoft Visual Studio 2008 - Microsoft.NET Framework 3.5 - Microsoft Windows 7 SDK - Microsoft Windows Presentation Foundation (WPF) - Web Services - Bing Translator API Character and Speech recognized technology - Microsoft Speech API (SAPI) Face recognition technology - Open Computer Vision Library 4
User Interface Prototype/Examples Figure 2: Hearing-impaired student use eyefeel to perceive the conversation. Figure 3: Conversation balloon pop-ups beside communicator face and conversation log at the bottom. 5
Figure 4: Multi-conversation balloon with face recognition system. Our Future work Figure 5: Client-Server application, Teacher speaks to server side and server send text stream to client which student can interact with it individually. 6