Title of Project: Soundless Music Team Members: MaryPat Beaufait, Benjamin Dennis, Taylor Milligan, Anirudh Nath, Wes Smith I. Introduction and overview of project a. Project Overview: Our project will entail the design and creation of a Soundless Music device which will enable the hearing impaired to experience music. The device will convert audio input to signals which can be recognized by the sense of touch. The project will be completed by April 12, 2012 for the College of Engineering Design Expo. The final product will consist of a sleeve with vibration motors, connected to an FPGA board (on which we will utilize DSP techniques) which takes audio input from an mp3 player or microphone. In our final presentation, we will demonstrate the conversion of arbitrary audio input signals (such as musical signals) to vibration patterns of motors mounted to a forearm sleeve. We want to accomplish this with as little time delay as possible. b. Project Justification: Music is an integral part of today s society. Music and its profound effects can be seen in virtually every culture, socioeconomic class, and age group. Unfortunately, not everyone has the ability to experience music due to disabilities affecting the sound organs. Our goal is to develop a device which will enable the hearing impaired to enjoy music though the use of vibrations. c. Potential Uses: There are many directives for the implementation of Soundless Music. For example, a concert hall could provide our device to hearing impaired customers. Parents of hearing impaired children could use the device to stimulate their child s cognitive development. The device might also be used on a day-to-day basis for recreational purposes. d. Digital Signal Processing Concepts Frequency Detection: Sampling, Fourier Transform (DFT, FFT). According to (6), the more data used for the FFT, the smaller the bins, or frequency intervals. Smaller bins would enable us to distinguish between individual frequencies with better accuracy. Of course, there is a trade-off. Using larger FFTs means: 1) we need more audio data, which could add delay to our system, and 2) it will take more time to run the FFT, which is CPU intensive by nature. Filtering: We will implement filtering techniques on the audio input signals to filter out noise and undesired frequencies. D/A and A/D Conversions: We will be using PMOD interface boards for data conversion (in addition to code as necessary). 1
e. usage We plan to use the Altera DE2-70 FPGA board for our digital signal processing. We will use the DE2-70 board for filtering, signal manipulation, and Fourier Transform calculations. Although our goal is to use only the FPGA board, we will consider using the C5515 ezdsp stick and its Fast Fourier Transform (FFT) libraries if the verilog FFT implementation isn t adequate. We also plan to utilize PMOD interface boards such as the PMOD DA1, DA2, AD1, and/or AD2. The FPGA board will interface with the vibration motors through an intermediate circuit. The vibration motors will interface with the forearm sleeve (vibration motors and sleeve will be purchased online). PMOD boards and FPGA board will be provided from the lab. II. Description of project i. The Soundless Music device will have the ability to convert an arbitrary audio input signal to non-aural sensory information. Objectives: To design and create a wearable Soundless Music device o Potential form factors include sleeves, gloves, or vests (prototype will be a single sleeve) o Non-intrusive device with minimal bulk o Minimized delay associated with conversion processes for optimal usage o To have a functional prototype by April 12, 2012 In order to determine the feasibility of our project, we researched and found projects of a similar nature that had been successfully completed. Details follow. In recent times several advances have been made in fields related to Soundless Music. Ryerson University has put significant research into the field and has come up with a few devices based on converting sound into vibrations which can be sensed by the skin. One such device is the EMOTI-chair, which has different vibration motor clusters throughout its body. Each cluster is designated for different instruments. Based on audio information such as volume and frequency, the various motors vibrate in ways meant to mimic the sounds. The device was created in November 2009 and it was used during a live concert for the hearing impaired. The EMOTI-chair proved to be a huge success among the attendees. (3) Dr. Dean Shibata, while working at the University of Rochester School of Medicine, conducted a study which examined the neural response that deaf people experience due to vibrations on their skin. He found that when deaf people feel vibrations on their skin, the region of the brain normally associated with hearing is activated. He claimed that his findings, suggest that the experience deaf people have when 'feeling' music is similar to the experience other people have when hearing music. (2) Researchers at the National University of Singapore published a paper titled, An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair. This chair, unlike the EMOTI-chair, was designed to amplify the natural vibrations caused from music. In the course 2
of this study, they discovered that...the Haptic Chair is capable of substantially enhancing the musical experience of deaf people, both children and adults. (7) In 2009, the Center for Entrepreneurship at the University of Michigan and the Deaf Performing Arts Network sponsored a competition where groups designed devices that allow hearing impaired people to experience music. The winning team produced a vest with 12 vibrating motors that respond to frequency changes of an audio input signal. The competing groups were awarded a total of $12,000. The fact that this event was sponsored proves that there is some market for a device of this nature. (1) ii. Figure 1, below, contains a block diagram of our anticipated design. Figure 1. Block Diagram The Soundless Music prototype device will accept input from either an external microphone or an mp3 player via an audio jack. For the microphone, the audio signal will require A/D conversion before it can be processed on the FPGA board. The mp3 player will deliver data to the FPGA in digital form. On the FPGA board, all digital signal processing techniques will be implemented. The signals will first be filtered to remove excess noise. Fourier analysis will then be performed. Code will be written to analyze both the frequency and intensity of audio at any given point in time. These findings will then be translated to vibration commands for each of the individual motors. These commands will be converted back to analog signals, and will be relayed to the vibration motors. 3
iii. Table 1 contains a risk analysis for the development period of the Soundless Music prototype. Table 1. Risk Analysis Item Number Observation Threat-Source/ Vulnerability Existing Controls Likelihood Impact Risk Rating Recommended Controls 1 Project Approval Design does not meet required DSP project specifications Similar past projects Low High Low Team preparations for project proposal 2 Unsuccessful MATLAB simulations Unable to use Fast Fourier Transforms EECS 451 course work Low Medium Medium Early preparations and start 3 Verilog FFT calculations 4 Prototype not functional Failure to translate MATLAB code to Verilog Interfacing issues between FPGA board and motor None Medium Medium Medium Use C5515 board instead None Low High High Early start and resource utilization 5 Time Delays Data conversions None High Medium Low Efficient conversion algorithms We have defined stages for our project (please see Milestones and Contributions discussions below) so that we always have something functional to fall back to in the event that any of the above obstacles are encountered. Milestone 2 was designed in such a way as to virtually guarantee the delivery of a fully functional prototype by April 3. This gives our team sufficient time to expand functionality as we wish, while still having a baseline prototype to demonstrate at the showcase. We plan to order equipment as soon as possible in order to test it and ensure that it will meet our project s needs. We will strive to use solely the FPGA board for the DSP, but we will use two boards (FPGA and C5515) if necessary. iv. Below we detail a preliminary parts list for our Soundless Music project. The available lab equipment for our project includes the C5515 ezdsp USB Stick development tool and the Altera DE2-70 FPGA board. We will also have use of the oscilloscope, function generators, and PCs available in the design lab. Our plan is to use microphones and an ipod for audio input, the Altera DE2-70 FPGA board for digital signal processing, PMOD A/D and D/A converters for data conversions for interfacing, vibration motors, and an UnderArmour sleeves. We plan to use the PMOD A/D and D/A converters in conjunction with the FPGA board. (About $30 each if not available in lab) 4
We will need to purchase the following equipment items: Omnidirectional microphones (5 at $1.05 each) Digikey product Vibration motors (12 at approximately $5 each) http://www.sparkfun.com/products/8449 Under Armour sleeves (2 at $20 each) http://www.underarmour.com/shop/us/en/mens/accessories/sleeves III. Milestones a. Milestone 1 (achievable by March 15). Milestone 1 is to have a fully functional MATLAB implementation. We want to load the audio signal file into MATLAB (by converting.mp3 into.wav), scan and sample the signal, and distinguish between frequencies using the Fourier Transforms. We would like our MATLAB implementation to be able to display the frequencies graphically. We will need to decide on a range of frequencies to represent, depending on frequency ranges associated with hearing and the frequency ranges of the sensors. During the time before the first milestone, we also want to have access to all hardware we will be using. In order to decide on our output frequency range, we need access to the sensors. As stated previously, we will order these as soon as possible. In the time before Milestone 1, we will be working on both the MATLAB coding and acclimating ourselves to the hardware and interfacing components. b. Milestone 2 (achievable by April 3). Milestone 2 is a presentable, fully functional device for the showcase. This entails audio signal data conversion, signal input to the FPGA board, digital signal processing implementation on the board, data conversions, and finally integration with the five vibration sensors on an UnderArmour sleeve. From ipod/microphone to sleeve, we want this to be complete by Milestone 2. c. Identify the one thing that scares you the most that could cause you not to achieve Milestone 1? How about Milestone 2? The one thing that scares us the most about achieving Milestone 1 is our code/mathematical implementations not working in MATLAB because of our lack of programming experience. No team members have taken programming courses beyond EECS 280, so our concern is that programming depth might hold us back. 5
The one thing that scares us the most about achieving Milestone 2 is again our coding proficiency. Our interfaces will require extensive programming, so our concern is our lack of experience, particularly in C and Verilog. IV. Contributions of each member of team The members of our team possess similar strengths and past technical experience. Our team consists of four EE majors and one CE major. Anirudh, who is majoring in CE, has the most practice with programming in our group because of his experience in EECS 373, EECS 370, and EECS 478. Taylor has some experience in embedded programming from EECS 461, and Anirudh and Wes are currently enrolled in EECS 461 as well. Ben and Wes have some experience with digital wireless communication from EECS 455, which may come in handy for a later iteration of our device should we decide to make the vibration motors connect wirelessly. Ben, MaryPat, Taylor and Anirudh have Verilog experience from EECS 270, which will certainly prove useful in programming the FPGA board. Reaching our first milestone will require MATLAB programming, and some knowledge of hardware and software integration. Our plan is to separate the MATLAB implementation into smaller components, such as filtering and ffts, and divide that work equally amongst the five of us while we wait for our hardware parts to arrive. Once all of the components of our device have arrived, we will split into two groups. One group will work on finishing the MATLAB implementation, and the other will work on building our device and testing the integration of the hardware and software. To reach the second milestone, we will have to be very efficient with how we divide the work. With the MATLAB simulation finished, it will be the first group s job to determine what modules or functions will need to be written to complete our project, while the second group shares the results of their investigations into the integration of our device. With a comprehensive list of what code will need to be written, we can then divide the coding as equally as possible based on past programming experience. Our second milestone should not be difficult to reach if everyone does his/her share, but is also willing to assist those who may be falling behind. Our current plan is to meet each week on Saturday at 1PM in the Duderstadt Center for progress reports and to work on anything that will require a collaborative effort. This will continue until March 5, 2012, when we will begin meeting 3 times per week. We will meet at the following times: Monday at 6PM, Wednesday at 6PM, and Saturday at 1PM. We are using Google Docs to collaborate and edit all necessary documentation, and we have an email group (soundlessmusic@umich.edu) for easy communication between all team members. 6
V. References and citations 1) Umich Feel the Music Competition links: http://ns.umich.edu/new/releases/7093 http://www.eecs.umich.edu/ece/awards/students/teams/pages/feelthemusic.html 2) WebMD showing that there is some medical background that suggests there is evidence supporting this idea: http://www.webmd.com/news/20011128/deaf-people-can-feel-music 3) EMOTI-CHAIR http://www.psych.ryerson.ca/mmm/emoti-chair.html 4) Music for the Deaf (neckset allowing deaf to feel music) http://bigthink.com/ideas/20311 5) Speaker allows deaf to feel music http://news.bbc.co.uk/2/hi/uk_news/england/london/4377428.stm 6) Fast Fourier Transform info http://bonkel.wordpress.com/2010/03/03/frequency-detection-using-fourier-transform/ 7) An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair http://delivery.acm.org/10.1145/1520000/1518756/p337- nanayakkara.pdf?ip=141.213.67.154&acc=active%20service&cfid=64833724&cftoken=315 02102& acm =1328381859_62a974e3cd9125b2f5873ef43bb8b05f 7