A Comparison of the Evaluation of the Victorian Deaf Education Institute Real-time Captioning and C-Print Projects

Transcription

1 A Comparison of the Evaluation of the Victorian Deaf Education Institute Real-time Captioning and C-Print Projects Final Report May 2016 Prepared for the Victorian Deaf Education Institute by Renée Punch PhD 1

2 This report provides a comparison of the evaluation findings of the 2015 C-Print trial as reported in the Evaluation of the Victorian Deaf Education Institute C-Print Trial Final Report 2016 with the evaluation findings of the real-time captioning trial as reported in the Evaluation of the Victorian Deaf Education Institute Real-time Captioning Pilot Program Final Report Similarities and differences between the C-Print system and remote real-time captioning C-Print is a speech-to-text captioning technology and service developed at the National Technical Institute for the Deaf (NTID), Rochester Institute of Technology, New York. The system provides communication access to individuals who are deaf or hard of hearing (D/HH) in educational and other environments. C-Print captionists, who may be present in the classroom or in another location, provide a record of what is said by instructors and students during a class. The program involves some summarization; the captioning text should be thorough and comprehensive, but is not intended to be a word-forword transcription. The text is available within a few seconds for students to read on their ipad or computer screens, and as a transcript to be accessed after the lesson. NTID provides an online education program to train captionists in the C-Print abbreviation system and text-condensing strategies. Remote real-time captioning (RTC), as utilized in the Victorian Deaf Education Institute (VDEI) RTC trial, also provides a text version of classroom speech to students computer or ipad screens. The aim of RTC is to provide a verbatim, word-for word transcription. The teacher s speech, captured by a lapel microphone, is relayed via the Internet to a remotely located captionist. This person immediately re-speaks it, with punctuation, into speech recognition software that converts it into text, which the captionist can correct if necessary. The text is then streamed to the student s device where it appears with a delay of about five seconds. The transcript of the lesson is delivered to the student within 24 hours. The stated goals of VDEI s use of both programs are to improve students access to curriculum material, improve students inclusion in the learning process, and enable greater communication and participation with peers and educators in the classroom. The aim of the evaluation studies of both programs was to assess the program trials in terms of their implementation and in terms of these educational and participation outcomes for the students who are D/HH involved in the trials. Similarities and differences between the RTC trial evaluation and the C- Print trial evaluation The evaluations both employed a mixed-methods approach combining the collection and analysis of both quantitative and qualitative data. This approach was underpinned by the framework developed by the U.S. Centers for Disease 2

3 Control and Prevention, The Framework for Program Evaluation of Public Health Initiatives The evaluations were based on the following data sources: a school data audit of students achievement, behaviour, and attitude over the course of their involvement with the program; standardised literacy assessments; student surveys; teacher surveys; and interviews with program staff. All of these data, except for the interviews, were gathered at two time points: early in the program, and again when the program had been running for some time. Full details of methods and data sources can be seen in the Evaluation of the Victorian Education Deaf Institute Real-time Captioning Pilot Program Final Report 2014 and the Evaluation of the Victorian Deaf Education Institute C-Print Trial Final Report The major difference in the evaluations of the two programs stems from the fact that the C-Print trial was considerably smaller in student numbers and briefer in its timeframe than the RTC trial. Whereas the RTC evaluation involved 24 students in six schools, the C-Print evaluation involved five students in three schools. It was intended that the C-Print trial would begin In Term 2 of 2015 and run for a nine-month period. However, due to staffing issues, it was not fully implemented until Term 3. Eight students received C-Print services in four schools. Because of staffing and timetabling issues and because one student chose not to take part in the evaluation, only five of the eight students were included in the evaluation. These five students were in three schools and received C-Print captioning in 80% of their classes in two subjects. The evaluation study covers this six-month trial of the C-Print program in Terms 3 and 4, Students in the RTC program trial also received captioning in 80% of classes in two subjects. The RTC trial evaluation reported on the whole school year of 2013, thus having a longer timeframe in which to consider changes in student achievement. It is therefore difficult to make direct comparisons of the trials in terms of student outcomes. Nevertheless, it is possible to give some indication of comparative trends between the findings of the two evaluations. Comparative findings between the RTC program and the C-Print trial Program implementation Unlike RTC systems that use their own off-site captionists and speech recognition technology, the C-Print system necessitates the employment in the schools of captionists with specialised C-Print training. For the initial setting up and implementation of the C-Print program, coordinators reported some difficulties with recruitment of captionists, and captionists reported some challenges with the training process. However, these had been resolved quite quickly. 3

4 From the survey and interview findings it appears that the C-Print program had slight advantages over RTC in terms of staff time and effort. In the C-Print evaluation study s interviews, coordinators reported more difficulties with the setting up of the RTC program, a greater workload with coordinating the RTC program than the C-Print program, and more technical problems with the RTC program. However, as the schools involved in the C-Print trial had previously participated in the RTC trial, it may be that staff members, having already become used to the challenges involved in setting up and implementing the earlier RTC program, found the C-Print implementation easier than it might otherwise have been. The interview findings in the RTC evaluation indicated that, although the initial period of implementation of the program involved considerable challenges and technological difficulties, these were able to be resolved and the process became much easier to manage as time went on. Student outcomes There are few definitive differences between the findings of the two program evaluations in terms of student outcomes. In both evaluations, teachers indicated a particularly positive response to the program and strong perceptions of benefit to the students involved. Students in both trials indicated a less enthusiastic but nevertheless generally positive response to the program. Students and teachers indicated similar levels of comfort with having the RTC captioning and the C- Print captioning in their classes. The evaluation studies findings indicate that both RTC and C-Print provided increased access to classroom instruction and discourse for participating students. Students from both program trials reported that they found the captions accessible in class in terms of being easy to read and delivered at the right speed, and that captions helped them to understand what the teacher was saying. Some students were using the transcripts after class for homework or revision purposes. Students in both programs indicated mostly positive responses to being involved in the program, with a slightly higher satisfaction rate among the C-Print students. At Time 2, 80% of students in the C-Print trial indicated that they wanted the captioning to continue in their classes and to be extended to other classes. For the RTC program, 64% of respondents wanted captioning to continue and 55% wanted it to be extended to other classes. Literacy assessments conducted at two time points in each program trial produced slightly better gains for the students in the RTC trial than the C-Print trial. For the RTC program, there was a five-month interval between assessments; for the C-Print trial, the interval was slightly less than three months. The short test-retest interval and the small sample sizes, particularly in the C-Print evaluation, mean that these results are preliminary and any changes cannot be attributed to the captioning program. A major aim of using a captioning service, whether RTC or C-Print, for students who are D/HH is to enhance their participation in the class and enable better communication with their peers as well as their teachers. For this to happen it is important that the D/HH student has access to whatever is said by other students during the class. In classes using C-Print the captionist was present in 4

5 the room and so could hear and transcribe everyone s speech, meaning that students questions and comments and general classroom discussion could be captured in the transcription. C-Print captionists reported that, as well as capturing the teacher s words, they tried to include as much of the class discussion that they could in their transcription. In both programs evaluation studies, teachers reported that the captioning provided D/HH students with an increased access to and sense of inclusion in class discussion. As a meaning-for-meaning captioning service whose aim is not a verbatim transcription of the speaker, the C-Print system involves some decision-making on the part of the captionist about how to summarise the spoken material. The captionist can leave out repetitive or inconsequential speech, and the ensuing transcripts are then briefer and more concise. Participants in the C-Print evaluation reported that this was occurring. However, with the C-Print system, D/HH students miss out on some of the class communication and thus are unlikely to have full access in the same way that normally hearing students have in the classroom. There is also a considerable responsibility for C-Print captionists to make the best choices about what to include and what to exclude in their captioning. In the C-Print evaluation interviews, one captionist reported simplifying the speaker s language to be more comprehensible to a particular student. Such judgements may not be appropriate to the role of captionist. With RTC, because the teacher wears a microphone, any quiet, private conversations the teacher may have with staff members or individual students are likely to be picked up by the captionist and included in the transcript. The RTC evaluation reported that some teachers found this to be a bit of an issue because they needed to remember to turn the microphone off for these interludes and then on again afterwards. Alternatively, they needed to instruct the remote captionist when to stop captioning and when to start again. This did not appear to be an issue with the C-Print program, presumably because the captionist, being present in the class, could easily tell what was appropriate to include or omit from the transcription. In both program evaluations, teachers reported that captioning was more effective for some students than others. In general, students with high levels of academic motivation and, in particular, literacy skills appeared likely to benefit the most from captioning. Conclusion Overall, the evaluation trials found that both programs offered benefits to students in terms of enhanced access to curriculum material, inclusion in the learning process, and communication and participation in the classroom. Both programs appeared to be well-received by teaching staff and the majority of the students involved. Although there were some differences involved in the implementation of the programs, neither program seemed to present major difficulties or challenges that could not be overcome. 5