TENT award report Evaluation of voice recognition software to produce automatic lecture transcripts and live subtitles during lectures

Size: px

Start display at page:

Download "TENT award report Evaluation of voice recognition software to produce automatic lecture transcripts and live subtitles during lectures"

Cecilia Harrell
6 years ago
Views:

1 TENT award report Evaluation of voice recognition software to produce automatic lecture transcripts and live subtitles during lectures Dr Kai F. Hoettges, Medical Engineering, FEPS Summary This project was originally conceived to used voice recognition software to deliver complete lecture transcripts, as well as live subtitled to lectures in order to support hard of hearing students as well as students of non-english speaking background. The project had to be scaled back to a feasibility study for technical reasons. Materials were not made available to the students, but some valuable insights were gained, that will enable to take the project futher at a later stage. In general voice recognition software can deliver a reasonable transcript, but a substantial amount of training is needed to get to this recognition rate. In particular technical terms are often not recognised without training and thereby led to inaccurate transcripts. Punctuation was not captured well, so the transcripts were badly structured and needed significant editing to be readable. Transcripts also demonstrate that spoken language varies widely from written language, so transcripts are not as easy to read as normal written text. Using the software increased my awareness of pronunciation, speed and sentence structure. This proved to be a very useful feedback to judge clarity of the lecture. The near instant feedback made the software a useful monitoring tool to improve my delivery and might be valuable as a training tool for public speaking. The relatively low spec netbook used during this evaluation struggled at times to process the lecture quickly enough to keep up, and computer speed should be considered in future trials The lack of structure in transcript makes the software more appropriate for either live subtitles or as a training tool for the speaker since document structure is less important in these cases. Original Concept Originally the project was conceived to improve support for a profoundly deaf student. Although the student was supported by disability support through various means (mostly note-takers), feedback from the student was that notes can be sparse and since the note takers are not trained on the subject matter, notes can be wrong or miss key aspects of the lecture. Therefore it was proposed to use voice recognition software to capture lecture and deliver a complete transcript. Since the voice recognition software works with a relatively short time delay it could also be projected during the lecture to give

2 real time subtitles and they allow more interaction with hearing impaired students. While the primary reason for this project was to help a single profoundly deaf student, background research suggests that such a system can have benefits for a wider section of the student population. Hard of hearing students are an obvious group benefitting from this technology and a substantial number of studies are registered with disability support for this. However John Beaumont (The universities disability coordinator) suggested that in the current IPod generation there might be a large group of students with undiagnosed hearing damage that are currently unsupported. Background research also suggests that non English speaking background 1 will also strongly benefit from a subtitles system since it helps them, to extend their vocabulary. The impact on other students with other disabilities such as dyslexia has not been assessed in the past, but there might also be benefits for students with learning disabilities. Software based voice recognition has been considered in the past 2,3, and it was concluded that software was unsuitable to capture lectures, since punctuation needed to be explicitly dictated and recognition rates were too low. However the only systematic studies were performed in 1998 and Software performance has improved since, and software manufactures claim recognition rated of >90% and automatic punctuation. These developments indicated that it may be worth reconsidering the technology. The ideal outcome of using voice recognition would be a better integration and progressing of hard of hearing students. The addition information my also be beneficial to the whole students cohort by aiding revision or helping with better engagement with the class. E.g. the slight delay in subtitles may help to pick up information that was missed because a student was briefly distracted or transcript may aid revision. Since a subtitles system could be integrated al low cost into existing IT infrastructure it could be developed into a general purpose teaching tool across lecture rooms. In the original proposal, the results were to be evaluated by student questionnaires; however, delays in purchasing the relevant software meant that the project had to be scaled back to a smaller feasibility study that was not made available to the students. However the project has still led to some valuable insights that can be used to continue the project once teaching resumes. 1 S.Bennett, J.Hewitt, B.Mellor, C.Lyon,,Critical Success Factors for Automatic Speech Recognition in the Classroom; Lecture Notes in Computer Science, 4556, (2007) 2 M.Wald, An exploration of the potential of Automatic Speech Recognition to assist and enable receptive communication in higher education; ALT-J, Research in Learning Technology, 14, 9 20 (2006) 3 Wald, M.: Developments in technology to increase access to education for deaf and hard of hearing students. In: Proceedings of CSUN Conference Technology and Persons with Disabilities. California State University Northridge (1999)

3 Feasibly study Dragon NaturallySpeaking 10 wireless was chosen as a suitable software package, since it can process speech close to real time and includes a wireless microphone to allow recording during lectures. Since full assessment in front of students was not possible due to late arrival of the software package, two approaches were taken for a feasibility study. A version of NaturallySpeaking that works only with dictaphones was available earlier so some lectures were recorded and processed afterwards. Recognition rates in the trials were too low to yield useful transcripts. A likely cause is that the microphone was not sufficiently directional and therefore picked up too much background noise. The software indicated low signal to noise ratios that might have affected recognition. Voice recognition software needs to be trained to each individual voice by reading example text to the software. The software also refines its recognition constantly by evaluating manual corrections. The training and correction program seems less sophisticated in the Dictaphone version that in the full version, leading to lower recognition. Due to these drawbacks the recorded lectures delivered no useful result. The full version of the software arrived only during the final week of teaching, and training could not be performed in time to use it during lectures. Therefore the full version of the software was assessed in mock lectures without students present. This led to valuable insights but conditions were not entirely realistic. The full software version has a more user friendly training interface, using a karaoke like interface, highlighting which words in the text it understood. If disturbed by too much noise the software does not move on until the sentence is repeated in adequate quality. This instant feedback helps to get better results. Also the supplied microphone seems to be more directional and picks up a good signal. After initial training the software performed still poorly on scientific lectures. Since it uses a statistical model to interpret the context of words, misunderstanding a key word in sentence can render a whole sentence nonsensical. This happened a lot during the first brief trial where key aspects of the lecture were not recognised. However the software allows correcting these errors manually after dictation and uses these corrections to improve recognition. After dictating three short lectures, the text overall made sense and was readable, though some errors still occur. However correcting these examples was very time consuming. Dictating short texts (5 min) and then correcting them seems to be the quickest way to improve recognition. The feasibility study also researched groundwork to impellent a live subtitle system. Previous studies found that the environed needs to be clearly structured to have the best impact 4. If an environment has too many visual focal points, it leads to visual dispersion. This means hard or hearing student need to juggle his attention between points of interest such as handouts, lecturer, subtitles, power point slides, whiteboard etc. This can lead to the student missing crucial information and tire rapidly. To minimise this visual 4 A.C. Cavender, Using Networked Multimedia to Improve Academic Access for Deaf and Hard of Hearing Students, General Exam Document, University of Washington, 28 January 2008,

4 information needs to be consolidated into a smaller space: Ideally a single screen that contains all this information. In real life lectures it is difficult to include the lecturer and whiteboard as well as electronic media into one device; however care should be taken that the subtitles are projected close to the main presentation screen. Ideally subtitles would be placed between the main projection screen and the whiteboard, to keep focal points as close as possible. The lecturer should stand close or underneath the subtitles if possible. However this needs to be experiments with in each separate location to identify the optimum layout. For initial trials a separate computer projector is suggested for the voice recognition. An independent system can be set up without interfering with the main presentation equipment in case errors occur. A trolley based system could enable faster setup/breakdown since there are usually only a few minutes turnover in busy lecture rooms. E-learning software that can place several streams of information in parallel on a single screen may deliver a good all round package with a high degree of integration A key parameter that needs to be investigated in class is the amount of disruption that subtitles may cause. If the software produces inaccurate transcripts it may actually be a distraction to the majority of students. Instant feedback after a first trail needs to be collected form the cohort to make sure it attention in the general class is not severely disrupted. If the majority of students is disrupted but hard of hearing students find a significant benefit, subtitles could be presented on laptops to a smaller sub group of students. In that case visual dispersion needs to be considered again in greater detail. Outcomes Through this initial study has no student feedback some conclusions can be drawn at this stage. Transcript quality Voice recognition delivered a reasonable transcript quality, but needed a significant amount of training to achieve this. The software uses a statistical language model to recognise which words are likely to appear in the context of other words. If the software does not recognise key words in a sentence, the whole sentence will be misinterpreted. Scientific terms used in a particular lecture are not likely to be recognised without prior training. However, correcting transcripts after dictation, counts as further training. Correcting recordings from a realistic lecture improves results, since lectures are delivered at a different pace and intonation than text read off a computer screen. Therefore delivering several short bursts of lectures as additional training improved recognition rate significantly. However, this increased the preparation time substantially. The first attempts took much longer to correct than to dictate. Several hours of training were needed to achieve acceptable quality. Punctuation NaturallySpeaking claims to automatically recognise punctuation. In my attempts this was not the case. Punctuation was only found sporadically there seems to be no correction mode for punctuation so it seems further training is

5 not really possible. However adding in punctuation manually is much quicker than other corrections. This is hampering the use for transcripts but maybe less critical for live subtitles during lectures. Capturing spoken language Use of the software highlighted that spoken language differs substantially from written language. Spoken language carries some information in intonation, includes flow control devices (e.g.: anyway, OK, well, oh, now, let s say) as well as disfluencies and hesitations (e.g. oh, um, aem, no but) 5. Naturally speaking is designed to capture text for written documents and uses models for this. These models may be less suitable for the spoken environment of a lecture. Reading transcripts of spoken conversations is generally more difficult than reading text that was written deliberately. This limits the usefulness of transcripts, since they may be perceived as confusing without the non-verbal information that was available during the lecture. Focus on language Trialling the software also led to a surprising insight that may yield significant benefits. Using the software to capture lectures increased my awareness of language enormously. The software provided instant feedback on language quality. Clear pronunciation increases recognition instantly, while mumbled or fragmented sentences were not recognised. This made me focus much more onto pronunciation and a clearer sentence structure. While using the software I was more conscious of hesitations and disfluencies and tried to focus hesitation, by leaving a brief thinking break in between sentences. This seemed to improve the recognition of punctuation slightly. In general I felt my delivery became slower and clearer through the instant feedback. This clearer delivery would be a benefit to students even when the software is not used, since it becomes more and more automatic. However, I will need to spend more time on it to internalise this fully. There is however, a question if there is time to monitor this feedback in a real lecture situation where my focus would be distracted by monitoring the reaction of my class instead of my laptop. All together it could be a valuable training tool for public speaking and lecturing since it can generate and instant feedback and does not need an observer to provide this feedback. Computer performance During the trials the software was run on a relatively low spec netbook. At times the processing speed was too slow and forced brief brakes to allow the software to catch up. In particular long sentences needed a disproportionately long time to process leaving a gap of more than two sentences to complete analysis. This could be easily overcome by running the software on a faster computer in live situations. 5 D.R. Olson, N. Torrance, A. Hildyard, Literacy, Language and Learning:The Nature and Consequences of Reading and Writing, (book) Cambridge University Press (1985) ISBN-13:

6 Conclusions The initial feasibility study highlighted two possible applications: Voice recognition for live subtitles and as a feedback and training tool for lectures. In both of these applications the quality of the final document is not of critical importance. Full lecture transcripts would need significant editing to add structure to the text. This would include adding headlines and links PowerPoint slides or notes on the whiteboards. Video recordings of lectures may be better revision tools for students since they bundle all this extra information and need less editing. In live subtitles this context would be available to the students. Punctuation and structure of the text is also less critical in subtitles. I observed that subtitles on TV normally do not include punctuation. (In contrast to movie scripts contain a lot of structure and contextual information that is not used in subtitles.) A feedback tool for training purposes does not need any structure or punctuation either, since it is purely for rapid feedback and is likely to be discarded after brief analysis and correction, but is not made available to students. Future work The use of the system for live subtitles has not yet been evaluated in front of students. In a follow on project this could be evaluated. It would be beneficial but not necessary to have students with severe hearing difficulties in the cohort. Initially, the software could be run on a separate computer and projector to supply the subtitles. This approach ensures that failures of the voice recognition will not affect the presentation system. NaturallySpeaking works in near real time. If the font size is set large enough the text is easily visible on a projector. The software scrolls automatically while transcribing and could be left alone if necessary. In a second step it may be useful to integrate it with presentation software so it can be run on a single computer and projector. NaturallySpeaking can transcribe into most windows software that allows writing from a cursor. So software that can integrate an extra text window with PowerPoint may be available. Background research suggest that good practice for such a system is to keep all important visual focal points in close spatial proximity, since rapidly focusing at a number of different points such as power point, subtitles, whiteboard and lecturer tires the eye. To evaluate the physical setup may be changed in successive lectures. The project could be evaluated by a simple questionnaire to the whole class that will evaluate key parameters such as: - Are the subtitles helpful or distracting - Is the recognition quality sufficient to be helpful - Has the students used the transcripts and how useful it was - Which spatial arrangement was most useful, - Does the student has any hearing difficulties, learning difficulties is he a native English speaker

7 - How doe subtitles compare to other support measures such as handouts, podcasts, textbook references etc. The evaluation will establish how much particular groups (hearing disability, non English speaking background, learning difficulties) a benefitting form the system and if the general class is distracted by the system or shows benefits as well. More specific questions may be put to students with hearing impairments, or other groups aforementioned that are being likely to show specific benefits.

Director of Testing and Disability Services Phone: (706) Fax: (706) E Mail:

Angie S. Baker Testing and Disability Services Director of Testing and Disability Services Phone: (706)737 1469 Fax: (706)729 2298 E Mail: tds@gru.edu Deafness is an invisible disability. It is easy for