4 User-friendly information presentation

Transcription

1 The start of NHK s Hybridcast is one step in our work to enhance broadcasting services. At the same time, it is important to develop the means for all viewers, including the disabled, the elderly, and non-native Japanese speakers, to be able to receive and enjoy its content. We are conducting research on using information technology to deliver the benefits of broadcasting to all people in forms that are appropriate for them. In our research on user-friendly presentation of information, we have studied tactile and haptic presentation methods for visually impaired people. We evaluated a method to facilitate understanding of 2D information and a method to present 3D shape information by giving multipoint stimuli to fingers. For hearing-impaired people, we upgraded our Japanese sign-language CG translation system for weather reports. We refined the speech recognition technology used for closed-captioning of information programs and succeeded in improving the recognition rate to 95.9%. We also developed an algorithm that uses news scripts for auto correction of recognition errors so that closed caption services can be economically offered by local broadcasting stations. In our research on speech and audio signal processing for the elderly, we upgraded the equipment to adjust program sound at the receiver. We refined the speech synthesis technology toward the realization of automatic reading services of broadcast scripts and data broadcasting texts and started research on the speech processing technology to enable various emotional expressions. In our research on natural language processing for barrier-free services, we developed technologies to improve the efficiency of the rewriting process of the news conversion assistance system into easy Japanese for non-native Japanese speakers in Japan and continued with our research on technology to automatically convert programming into easy Japanese. We also developed a natural language analysis system to categorize viewers opinions on broadcast programs. In our research on content retrieval and recommendation, we built a content map connecting programs with various relations by analyzing program description texts and studied a method to give reasons for retrieving and recommending programs. The program retrieval technology has been incorporated into a Hybridcast application. We also researched Video bank to assist in the retrieval and processing for raw video footage by adding metadata useful for video production to the raw video footage. The analysis technology we developed was used for managing raw video footage at a local station. In our research on estimating the psychological state of viewers, we conducted experiments to clarify the cognitive characteristics and measure brain activities related to the sensation depth caused by high-resolution video such as Super Hi-Vision. We also continued with our research on the analysis of psychological states while watching video by measuring the brain activities of viewers and research on advanced technologies to prevent negative effects of video on the human body. 4.1 User-friendly information presentation NHK STRL is researching user-friendly information media that will allow people with vision or hearing impairments to get more information from broadcasts. Haptic display technology of nonverbal information We are researching technologies to convey information that is difficult to convey in words, such as diagrams and the shapes of artworks, to people with visual impairments by using 2D and 3D information that can be apprehended by the tactile and kinesthetic senses. Our work on tactile presentation of 2D information includes a method to convey an understanding of diagrams and graphs to people with visual impairments by vibrating a tactile display. With this method, we found that vibration per line segment in addition to vibration per facet can reduce the time required for people to grasp spatial positions in diagrams and line charts. For the powered mechanical leading method that conveys the overall composition of a diagram by guiding the fingers, we derived conditions that can be placed on the leading speed to reduce errors in spatial position recognition. We developed a method to present 3D information using a multi-point force presentation method that gives stimuli to the finger. We conducted subjective evaluations with visually impaired people and objective evaluations using the length Figure 1. Multi-point force presentation method 22 NHK STRL ANNUAL REPORT 2013

2 4.1 User-friendly information presentation 4.2 Speech recognition for closed captioning of finger tracks as an evaluation indicator, in order to identify how the number of stimulus points affects cognition of the ridge line. The results showed that using four stimulus points is preferable (1). We also evaluated the effect of the degrees of freedom of the stimulus points when force is applied in the x, y and z directions on tactile cognition of curved surfaces. The results showed that at least one stimulus point needs to have three directional degrees of freedom. This result can be used as a design guideline for building presentation equipment. Part of this research was conducted in cooperation with the University of Tokyo. Sign-language CG translation technology for weather information We are researching technology to automatically translate Japanese weather reports (text) into sign language computer graphics. We continued our work from FY 2012 on improving the accuracy of translation by using example-based and statistical machine translation of clauses or phrases and upgraded our Japanese sign-language CG translation system. We prototyped a translation system in which the presentation speed of the sign-language CGs is adjustable so that the CG can be played in synchronization with the original weather report. The system was exhibited at the NHK STRL Open House. We also developed a system to release the Japanese and sign-language dictionary used in our translation technologies on NHK Online to allow anyone to access the service (Figure 2). This system will make the results of our studies helpful for people learning sign language and will make it easier for us to gather opinions on the quality of sign-language CGs and use them to make improvements to them (2). Part of this study was conducted in cooperation with Kogakuin University. Figure 2. Sign-language CG website (1) T. Handa, T. Sakai, T. Shimizu: Recognition of Three-dimensional Geometry using Multipoint Force feedback on a Fingertip, IEICE Technical Report, vol. 113, no. 347, WIT , (2013) (2) NHK Sign language CG Website, Speech recognition for closed captioning We are researching speech recognition for expanding the range of programs covered by real-time closed captioning so that more people including the elderly and those with hearing difficulties can enjoy TV programs At the end of March Addition of 150 million words Recognition error rate (%) Discriminative acoustic model Search optimization + Topic adapting language model Elimination of restatement Figure 1. Reduction of recognition errors in re-spoken speech for information program Closed captioning for information programs The re-speak method, in which automatic speech recognition is used on a speaker in a quiet environment who is repeating the speech of a program, is used to produce closed captions for sports programs such as sumo wrestling as well as information programs such as Asaichi. Closed captioning for information programs covering a wide range of topics requires a language model having word-sequence probabilities that change daily with each new topic and an acoustic model compliant with the changing frequency distributions of vowels and consonants of the re-speaker. These frequency distributions may change when the speaker becomes fatigued during the re-speak task. In FY 2013, we trained the language model by adding about 150 million words including closed caption texts for information programs with a broad range of topics. We also incorporated a discriminative acoustic model trained to take advantage of complementary searches with the language model and optimized the search method. We developed a method of training the language model by adapting it to topics (1). In an experiment, these efforts reduced the recognition error rate of re-spoken speech selected from the Asaichi program from 7.3% to 4.1%. Closed-captioning for news programs We helped to install speech-recognition closed-captioning systems for news programs (2) in the NHK Osaka, Nagoya, Fukuoka and Sendai stations. These systems are being used for closed captioning of local programs. We also developed an al- NHK STRL ANNUAL REPORT

3 4.2 Speech recognition for closed captioning 4.3 Speech and audio signal processing for the elderly gorithm to automatically correct recognition errors by using news scripts. This algorithm will be implemented in a closed captioning system at smaller local stations (3). (1) Y. Fujita, T. Oku, A. Kobayashi, S. Sato: Language Model Adaptation by Constrained Non-negative Matrix Factorization, Spring Meeting of the Acoustical Society of Japan, (2013) (in Japanese) (2) A. Kobayashi, Y. Fujita, T. Oku, S. Sato, S. Homma, T. Arai, T. Imai: Live Closed-Captioning System Using Hybrid Automatic Speech Recognition for Broadcast News, NAB Proceedings, pp (2013) (3) S. Sato, K. Onoe, A. Kobayashi, T. Oku, Y. Fujita, M. Ichiki: An error correction algorithm by using a WFST built from news scripts, Spring Meeting of the Acoustical Society of Japan, (2014) (in Japanese) 4.3 Speech and audio signal processing for the elderly To provide easy-to-hear audio services for everyone including the elderly, we are researching technologies to adjust program sound at the receiver and to synthesize good-quality speech. Earlier, we had developed a technology to adjust the balance of speech and background sound arbitrarily (1). In FY 2013, we made the algorithm less computationally intensive in order to make it more practical and developed equipment that incorporates the algorithm and speech rate conversion technology. Experimental evaluations showed the effectiveness of this equipment (Figure 1). In our research on high-quality speech synthesis and process technologies, we had previously developed a method to synthesize high-quality speech of arbitrary sentences by using massive amounts of speech data recorded from news programs. We also had developed automatic reading equipment and used it in broadcasts of the Stock market report program on NHK Radio 2. In FY 2013, we improved the speech synthesis technology and began study on speech process technology to convey emotional impressions and thereby expand the range of expression of synthesized speech. To improve the performance of speech synthesis technology, we modified the previous database method, which directly handles acoustic and linguistic features of speech data. By changing the search unit to strengthen the connection of unvoiced consonants and silences that have less auditory influence, we increased the sequentiality of phoneme strings segmented from text (2). To stabilize the synthesized sound, we studied a way to switch to statistical models built by using machine learning of features when no phoneme string in the database. In our work on speech process technology, we prepared an emotional voice database for adding emotional expressions to individual synthesized speakers. We collected speech judged to convey certain emotions and devised rules for adding emotional expressions to speech by analyzing the differences between the acoustic features of speech conveying no emotion and the features of emotional speech (3). We also developed a speech conversion technology that applies these rules. We conducted experimental evaluations that verified the emotional expression rules could correctly alter unemotional speech to a particular emotion 60% of the time. We incorporated our results in a system for automatically reading stock market reports. This resulted in lowering the minimum unit of nominal prices to sen (1/100 of one yen) from yen, extending the maximum readable number from less than 10 million to less than 100 million, and allowing a market overview to be read. We also worked on an automatic weather report reading system. We manually checked all the waveform extension and contraction units obtained from a prior analysis of speech data and corrected errors in order to avoid reductions in sound quality due to the speech rate conversion. The improved automatic stock market reading system began operations at the end of March (Figure 2). Figure 2. Automatic reading system Figure 1. Evaluation experiments (1) Komori, Imai, Seiyama, Takou, Takagi, Oikawa: Development of volume balance adjustment device for voices and background sounds within programs, for Elderly people, AES 135th Convention Paper 9010 (2013) (2) Segi: Search-unit selection for concatenative speech synthesis systems, Proceedings of 2014 Spring Meeting of Acoustical Society of Japan (3) Seiyama, Segi, Imai, Takagi: Evaluation of speech database for emotional speech synthesis, ITE Winter Annual Convention 13-4, NHK STRL ANNUAL REPORT 2013

4 4.4 Language processing for barrier-free services 4.4 Language processing for barrier-free services Japanese translation/conversion assistance technology We are researching technology to assist in the task of converting news program scripts into easy Japanese for non-native Japanese speakers in Japan. The NEWSWEB EASY test site started in FY 2012, which provides regular news text rewritten into easy Japanese, began its full-scale operation in FY Five news articles are released per day. To shorten the rewriting time, we developed a method to automatically assigning detailed information used for showing readings in furigana on top of kanji, displaying dictionary data for difficult words and highlighting proper names, achieving the accuracy of 95% or higher (1). We also began research in FY 2013 on methods to divide a news script sentence into several short sentences and convert modifier phrases in independent sentences (2) in order to automatically convert news scripts into easy Japanese. We prepared text data required for this study, such as examples of rewritten news scripts. Opinion analysis technology We are researching technology to analyze opinions of viewers of a program in order to utilize them in program production. In FY 2013, we developed an algorithm to automatically detect program names in Twitter messages about programs and built a system to analyze tweets in a structured and exhaustive way using the algorism (Figure 1). The official name of a program or a hashtag indicating a program name is not explicitly mentioned in most of the tweets about TV programs. So, we developed a method to identify which program is mentioned in a tweet. We focused on three elements: a program name including its abbreviation and the names spelled in different ways, program description, and expressions indicating viewing of the program, and devised a method to combine those three elements. This enabled detection of the program mentioned in tweets with 77% accuracy, even if the official program name or hashtag is not included (3). The program name detection algorism requires handmade rules to cover program names in abbreviation and different spellings. As these rules are used in combination with the algorisms to detect program description and expressions of viewing status, setting rules for the names of regular program serial will be enough to detect individual program name. This made the maintenance of rules easy and saved workload for continuous feedback analysis. (1) H.Kumano, H.Tanaka: Online learning of tagging to Japanese documents with dependent Dirichlet process, 20th annual convention of ANLP, pp , (2014) (in Japanese) (2) I.Goto, H.Kumano, H.Tanaka: Analysis on manual rewrite from normal Japanese news to easy Japanese news, 20th annual convention of ANLP, pp.15-18, (2014) (in Japanese) (3) M.Hirano, K.Kanbe, T.Kobayakawa: Automatic detection of TV program mentioned in tweets-for the overall and continuous detection-, ITE Annual Convention 2013, 3-7, (2013) (in Japanese) Results of structured and exhaustive analysis Documentary / Culture Cartoon / Special effects Movie News / Report Variety of programs Theater / Performance Hobby / Education Welfare Program A 421 tweets Program B 81 tweets Total number of tweets (1,322) Variety show Kids Program C 87 tweets Music Drama Information / Tabloid show Language Program D 733 tweets 0% 50% 100% Various tweets about TV programs Positive Negative Neutral Request Recommend Stand-by Viewing Plan Others Figure 1. analysis system NHK STRL ANNUAL REPORT

5 4.5 Content retrieval and recommendation technology 4.5 Content retrieval and recommendation technology Content utilization technology using language information We are researching technology that uses text information to manage, retrieve, and recommend broadcasting content. In our study on managing large amounts of content, we developed a method to identify the topics of a program and infer the relations between the topics and the program by using program descriptions in the Electronic Program Guide (EPG) (1). Using the obtained topics and relations and the semantic relations between words we had developed earlier, we built a content map to connect programs (Figure 1). The content map can manage programs with relation names, for example, treatment and prevention for a program with the topic of high blood pressure. We also developed a technology to extract relations between programs from viewer feedback about programs. Experiments with actual feedback demonstrated that the method can identify many useful relations that are not included in the program descriptions; for example, it can determine if the same singer s songs are used in two different programs. In our studies on retrieving and recommending content, we devised a method to extract words indicating the reason for retrieving and recommending a program from the program description (2). The effectiveness was experimentally demonstrated in tests on a week s worth of programs to be broadcast. The results of these studies are being utilized by the Hybridcast application Minogashi, natsukashi launched in December 2013 (Figure 2). Video bank for utilizing raw video footage To make use of raw video footage stored in video archives, we are researching a video asset management system, called Video bank, which uses physical sensor and video analysis technologies to automatically add metadata useful for video production. We developed a hybrid sensor consisting of small attitude and distance sensors created using semiconductor manufacturing technology to acquire camera motion information (camera parameters) required for video synthesis. We also improved its operability by adding error compensation functionality using pattern recognition. We also developed a method to acquire more robustand precise camera parameters that uses the hybrid sensor and a camera parameter estimation method that works by analyzing video footage. In our work on video processing technology, we developed a method to interactively extract a specified object region in video. We improved the usability of the object extraction function by preprocessing the region segmentation that requires a high calculation cost and by devising an extraction algorithm that takes into account the size of the pre-segmented regions to make sure an appropriate region matching the user s directions is extracted. Regarding our work on video retrieval technology, we made the discrimination of objects more precise by analyzing numerous feature points, the relations between these points, and their positions in a region of the image (3). We devised a technology for extracting key frame images containing an entire object by using the region segmentation method and the image features of individual regions. We also studied video retrieval technology using the similarity of images. Here, we improved the retrieval accuracy of cuts, which is the minimum shooting unit, by evaluation of the similarity of the main object regions. After that, we developed a technology to extract a series of cuts shot at the same place (i.e., scenes) by using the appearance frequency of similar pictures together with a technology to retrieve scenes. We used these video retrieval and object extraction technologies to improve our earthquake disaster metadata system. We also built an experimental system with functions for getting and processing video from archive tapes by adding and correcting metadata. The system has been used at the NHK Fukushima station since the end of September 2013, and it automatically added metadata that would enable retrieval-byshot to more than 10,000 video tapes related to earthquake disasters in only three months (Figure 3). Figure 1. Content map PC for retrieval Data storage disk Related programs Figure 2. The Minogashi, natsukashi Hybridcast application PC for processing LTO drive Figure 3. Earthquake disaster metadata complementary system being tested at NHK Fukushima station 26 NHK STRL ANNUAL REPORT 2013

6 4.5 Content retrieval and recommendation technology 4.6 Viewersʼ mental state estimation technology Part of this research was commissioned to NHK Engineering System, Inc. The research was conducted in cooperation with the Shimizu Corporation. (1) K. Miura, I. Yamada, T. Miyazaki, N. Kato, H. Tanaka: Generating a TV Program Map Using Semantic Relations between Words, In Proceedings of the 76th National Convention of IPSJ, 5C-4 (2014) (in Japanese) (2) T. Miyazaki, A. Matsui, I. Yamada, N. Kato, M. Naemura, H. Sumiyoshi: ICA based keywords extraction for TV program recommendation, In Proceedings of ITE Annual Conference 2013, 12-4(2013) (in Japanese) (3) Y. Kawai, M. Fujii: Semantic Concept Detection based on Spatial Pyramid Matching and Semi-supervised Learning, ITE Transactions on Media Technology and Applications, vol. 1, no. 2, pp (2013) 4.6 Viewers mental state estimation technology To understand how the viewers watch TV programs and how they are psychologically influenced by them, we are researching ways of estimating their mental state from subjective evaluations and brain activity. In FY 2013, we evaluated the sensation of the depth of objects in natural (i.e., not artificially shaded) scenes with varying resolution to explore the psychological effects of high-resolution images such as those of Super Hi-Vision. The results showed that the depth sensation increases as the resolution becomes higher, and this finding corroborates the results of our previous experiments in FY 2012 on artificially shaded images (1). To see what neural mechanism underlies the depth sensation, we measured brain activities when viewers make a judgment about the depths of objects in images of varying resolution. The results demonstrated that a specific brain area that is wellknown to be specialized for spatial and motion perception can be associated with the judgment of monocular depth (2). In our research on analysis of mental states of persons while they are watching video, we measured the brain activities of subjects while they viewed a program by using functional magnetic resonance imaging (fmri) equipment and analyzed them using machine learning in order to estimate changes in mental state through time. The results of an experiment on persons watching comedy TV programs showed that viewers are in a mental state of expecting the development of the program before they actually find it funny and that such a mental state is measurable (3) (Figure 1). To prevent video from having undesirable effects on the human body, we developed a method to estimate the degree of unpleasantness of shakiness, flicker and striped patterns in video and experimentally demonstrated its effectiveness. (1) K. Komine, Y. Tsushima, N. Hiruma: Higher-resolution image enhances subjective depth sensation in natural scenes, Perception, Vol. 42, supplement, p. 119 (2013) (2) Y. Tsushima, K. Komine, N. Hiruma: Cortical area MT+ plays a role in monocular depth perception, Perception, Vol. 42, supplement, p. 149 (2013) (3) Y. Sawahata, K. Komine, T. Morita, N. Hiruma: Decoding humor experiences from brain activity of people viewing comedy movies, PLoS One 8: e81009 (2013) Program video Continuous evaluation by slider Intensity of percieving humor Perception of Humor Brain activities Predict a future humor experience Time (sec) fmri Figure 1. Prediction of humor experiences from brain activity analysis NHK STRL ANNUAL REPORT