Effects of Music on Image Impression and Relationship between Impression and Physical Properties

Transcription

1 Electronics and Communications in Japan, Vol. 96, No. 9, 2013 Translated from Denki Gakkai Ronbunshi, Vol. 131-C, No. 8, August 2011, pp Effects of Music on Image Impression and Relationship between Impression and Physical Properties KEIKO SATO 1 and YASUE MITSUKURA 2 1 Ritsumeikan University, Japan 2 Keio University, Japan SUMMARY Auditory information plays an integral role in AV media because even identical images are perceived differently when they are matched with different music. However, we now present the results of experiments in which the changes in subjective perceptions were analyzed on the basis of the physical properties of the perceived items. The purpose of this study is to investigate the effects of music on image impressions in terms of the physical properties of images. In this paper, we first elucidate the changes in subjective impressions when the image is presented by itself and when it is presented with music. Second, to clarify the relationship between the impressions of images or music and the physical properties, we compare the different image or music perceptions with each other and also compare their respective physical properties, which include color information, structural information, and frequency characteristics. The color information of an image containing green or saturation colors and the power of the music are found to be strongly correlated with adjectives expressing potency and activity. The entropy of saturation is found to be correlated with words expressing spatial extent Wiley Periodicals, Inc. Electron Comm Jpn, 96(9): 53 61, 2013; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI /ecj Key words: music; image; impression; interaction; physical properties. 1. Introduction The combination of visual information and auditory information produces various effects. Auditory information such as music or sound effects has commonly been presented through AV media, so that we receive a high-impact impression. It is known that the impression received from a visual scene is greatly influenced by visual information. However, it is clear that auditory information plays an integral role and has a significant impact on the perception of a visual scene. That is, the combination of visual and auditory presentation produces a complementary effect. How different sensation modalities such as visual and auditory interact with each other is a subject addressed in the field of psychology. In the field of perceptual psychology, the interaction between the visual and auditory sensory modalities is well documented. For example, brightness represents both visual and auditory impressions. This psychological characteristic, common in some sensation modalities, is known as intermodality. It is also known that auditory brightness alters visual brightness in the same direction. This phenomenon is called consonance [1, 2]. Auditory information has a significant impact on the perception of visual information because even identical images are perceived differently when they are matched with different music. Previous studies of the impression received from combinations of visual and auditory information include experimental studies from the psychological field [3 5] and application studies on interretrieval systems among different media such as methods for the selection of music corresponding to pictures [6]. First, almost all experimental studies have investigated visual auditory interaction by the semantic differential method and factor analysis. For example, Iwamiya has shown that various effects are created by combinations of music and motion pictures from the emotional viewpoint [3]. Second, in application studies, this system has been based on common impression words among image or music and on the degree of coincidence. However, these studies did not consider cases in which different combinations of impressions match up with each other. Furthermore, if we wish to change the impression of an image to a different impression, it will not be possible to solve this problem. This will require analysis of the interaction based on the physical Wiley Periodicals, Inc.

2 properties of the image and music as well as study from the psychological viewpoint. There have been some previous studies that focused on physical properties in order to examine the interaction of the visual and audio effects characteristic of complex materials. For example, there is an analysis of the effects of color on the impressions of music [7]. In addition, several studies of audiovisual congruency have focused on physical properties. These studies have indicated that audiovisual congruency can be classified into two types: semantic congruency and formal congruency [8, 9], and have investigated both types based on physical properties. Semantic congruency is similarity between auditory and visual impressions, and it has been shown that musical and visual elements affect the perception of audiovisual content and that there exists semantic congruency of sound and motion pictures [10]. Formal congruency is the matching of auditory and visual structures. Related works include the effects of synchronization between auditory and visual accents and those of matching between musical tempo and visual speed [11], congruence between the rotating movement of a visual image and pitch modulation of a sound [12], and congruence between switching patterns of moving pictures and pitch shift of sounds [13]. Many investigations have focused on the physical properties of audiovisual congruency. However, there have also been a few studies of changes in subjective impressions when visual information is matched with auditory information, but no analyses based on the physical properties of the perceived items have been conducted. Thus, it is imperative that the relationship between the physical properties of images or motion pictures and human emotions be defined. The purpose of this paper is to elucidate the effects of music on image impressions considering physical properties. Specifically, we investigate the changes in subjective impression when images are presented without music and with music. We then compare the different image or music impressions with each other and also compare their respective physical properties. In this paper, we first describe an experiment on image impressions based on the semantic differential method. In this experiment, the participants evaluated image impressions when only the image was presented and when the same image was presented with music in the background. Next, we applied factor analysis technique to the data obtained from the experiment. Using data from the experiment, in order to reveal the relationship between the impressions of image or music and their physical properties, we calculated the correlation between the impression value and the magnitudes of the physical properties. For all of these analyses, we determined the effects of music on image impressions and the relationship between the impressions and the physical properties. This investigation is basic research on the audiovisual interaction. We believe that the result presented will be applicable to the automatic selection of music in AV media or psychological therapy. 2. Experiments In this section, we describe the stimuli and summarize the experiments. 2.1 Stimuli As the first step, general scenery images were used as visual information [14] and piano music as auditory information. The stimuli were very simple, and it was easy to extract the physical properties and identify the factors in impression formation. We used eight different images, as shown in Fig. 1 and Table 1, based on a preliminary questionnaire in order to prevent impression bias. In the preliminary questionnaire, 10 participants were asked for their impressions of some images, expressed as bright, or dark, or neither. We then selected three bright images, three dark images, and two neither images from among the images for which over 8 participants expressed the same impression. Similarly, for music, we selected four examples of bright music and four of dark music, for a total of eight musical examples as shown in Table Experimental summary In this experiment, the image ( pixels) was first presented without music. The participant expressed his or her impression of the presented image using a semantic differential scale [15] (impression evaluation words: warm cold, loose tight, clear cloudy, soft hard, active inactive, extended narrow, fancy plain, cheerful gloomy, pleasant unpleasant, new old, wet dry, calm excited, beautiful ugly, bright dark, light heavy), consisting of 15 adjectives with a 5-point bipolar scale: (+2: very warm, +1: warm, 0: not warm or cold, 1: cold, 2: very cold). These words were selected on the basis of related studies of image impressions [16, 17]. Next the same image was presented along with music that had been edited down to about 1 minute. The music was played in the combinations shown in Table 1 and was repeated until the participant had finished the evaluation. In addition, we used three dummy images in order to reduce the influence on the impressions of images that the participant had already seen once; the dummy images were not used for analysis. We changed the order for the participants to avoid the effects of presentation order. The participants were assigned the task of selecting impressions for 19 images in all, together with 3 dummy images. This experiment was conducted under ordinary lighting and took about 20 minutes. The stimuli were presented 54

3 Fig. 1. Images presented to participants. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.] using a personal computer and the participants wore headphones. The participants were able to adjust the sound volume to a preferred level. We did not use the data from participants who had made major changes in our analysis. The participants in the analysis were 20 students (11 males and 9 females). 3. Analysis of Changes in Impression by Factor Analysis We used factor analysis to investigate the changes in image impression caused by music. Using the data for 20 participants (the impression values when eight images were presented without music and when the same images were presented with music), we applied factor analysis (principal factor analysis, varimax rotation) to 15 impression words and extracted two factors. We show the factor loadings and the contribution rates in Table 2. The first factor includes many adjectives which express positive or negative light conditions and some adjectives expressing spatial extent or narrowness. We labeled this factor comfort. The second factor is marked by high loadings on potency items and impression words which include bright dark, cheerful gloomy, active inactive, and light heavy, so that it was labeled potency and activity. The factor score distribution is shown in Fig. 2, where the horizontal axis represents the first factor and the vertical axis represents the second factor. Each factor score represents an average value for 20 participants; the starting point of an arrow represents the score when only the image was presented, and the end point represents the score when the same image was presented with music. The results obtained from the scores when only the image was presented lead us to infer that images 1, 2, and 3 have positive scores for both factors. Image 7 has a positive score for the first factor. Images 4 and 5 have negative scores for the first factor, but Table 1. Images and music 55

4 Table 2. Results of factor analysis magnitude of the changes in impression. Both scores of images 5, 6, and 7 are higher with music. On the other hand, both scores of images 2, 3, and 8 are lower with music. The first factor score of image 4 is higher, while the second factor score is lower. The music with images 2, 3, 4, and 8 had a dark impression, and the music with images 1, 5, 6, and 7 had a bright impression based on the results of the preliminary questionnaire. Thus, bright music improved the image impression score, whereas dark music lowered it. However, we see the opposite for images 1 and 4. Music with image 1 lowered the comfort factor, whereas music with image 4 improved it. 4. Relationship between Impression and Physical Properties image 4 has a positive score for the second factor. Images 6 and 8 have negative scores for the second factor. Thus, the impressions of images 1, 2, 3, and 7 are pleasant, loose, beautiful, warm, and the like, while those of images 4, 5, and 8 are unpleasant, tight, ugly, cold, and the like. Furthermore, the impressions of images 1, 2, 3, and 4 include the ideas of fancy, new, and others while those of images 6 and 8 include the ideas of plain, old, and others. In the results obtained from the scores when the same image was presented with music, we can see that the first factor score of image 1 is slightly lower, while the second factor score is higher. We can interpret the line length as the We have already examined the relationship between image impression and the HSI histogram of an image [18]. In this section, we describe an analysis conducted in order to reveal the relationship between the impression of an image or music and their physical properties. 4.1 Physical properties of image In a perceptual process, it is known that there is a natural circuit that can perceive local or regional color, tone, and saturation. By coordinating these perceived items, we perceive the shape or texture of the object. An image is a combination of various items, which include line, color, and others. Based on these items, painting and photography project a general effect. The physical properties of the image considered in this paper are color information and structural information. Color information is expressed by mean values and entropies in the RGB value and HSI color space. In addition, we use a Gabor filter to provide structural information. The physical properties of the images used in this paper are shown in Table 3, and we describe each feature below. Table 3. Physical properties of images Fig. 2. Changes in image impression (mean value for 20 participants). 56

5 4.1.1 Color information For color information on the images, we use the mean values and entropies in the RGB value and HSI color space. By using not only the RGB value but also the HSI color space, it is possible to obtain characteristics such as human perceptual color relationships. The mean value is calculated by dividing the sum of the factors of the two color spaces by the number of pixels. The other color item of information, the entropy, is a measure of the uncertainty associated with a random variable. In this paper the entropy is a measurement of a color distribution that expresses amount of information that the distribution contains and is expressed as follows: where i is the information content and P(i) is the relative rate of occurrence of each of the n colors (in the case of hue, n = 360) in the image. Characteristically, the pixel histogram is broad and multimodal, with large entropy. For example, hue-entropy of image 4 is the smallest among the images used in this experiment, while that of image 5 is the largest Structural information We consider the direction feature as structural information. In order to extract the direction feature from the image, we use a Gabor filter. The use of the Gabor filter makes it possible to see whether the image is marked by straight lines or transverse lines. The Gabor filter is used to extract the edge properties by setting the proper parameters, modeling the response of human visual cortical cells. The directional selectivity of the Gabor filter is used as a feature extraction method in many image-recognition techniques, such as face and fingerprint recognition [20, 21]. Therefore, we believe that the physical properties obtained from the Gabor filter are appropriate in dealing with the image impression. The Gabor filter is the product of a Gaussian and a cosine function. In this paper, given an image, we convert the RGB color image to a grayscale image and pass it through a bank of Gabor filters as shown below. The impulse response of the Gabor filter F(x, y) is given by (1) Fig. 3. Image after Gabor filtering (θ = 135 ). four rotations (θ = 45, 90, 135, 180 ) to the eight images. Figure 3 shows image 7 after Gabor filtering (θ = 135 ). The parameters are as follows: σ = 5, λ = 4.2, γ = 1. We calculated the mean value for the images after Gabor filtering and used it for the physical properties of image. 4.2 Physical properties of music In order to investigate the relationship between the magnitude of changes in the image impression and the physical properties of the music, the frequency characteristic obtained by the FFT (fast Fourier transform) was used. We believe that this has the advantage of being able to extract the tonal features of different music. In this paper, we first cut out the 30 seconds at the beginning of the eight musical selections (44.1 khz, 16-bit, stereo wave), because it is known that we form first impressions of an object in a short time, only a few seconds. After this we transformed the eight musical selections (30-second music clips at the beginning) into spectrograms with a window size of 1024 samples. A Hamming window with 50% overlap was used and the FFT is calculated in each window. We next calculated the mean power at all frequencies (mean), the standard derivation (SD), the mean value at low frequency (MLFP), the mean value at high frequency (MHFP), the maximum power (max), and the frequency at the maximum power (frequency). Table 4 shows the physical properties of the musical selections considered in this paper. (2) Table 4. Physical properties of music where x = x cos θ + y sin θ, y = x sin θ + y cos θ In this formula, x and y are the image coordinates, γ is the aspect ratio of the filter, θ is the rotation angle, λ is the radial frequency, and σ is the spatial frequency. We applied only 57

6 Table 5. Results of correlation analysis (factors) where r and n are the correlation coefficient and the number of data, respectively. t 0 follows the t distribution with n 2 degrees of freedom. In this investigation we performed a two-tailed test at the 5% significance level. We calculated the coefficient of correlation between the impression values (15 adjectives) and the physical properties and tested for statistical significance. 4.4 Results and discussion 4.3 Correlation analysis We calculated the coefficient of correlation between the two factors and the above-mentioned physical properties in order to ascertain which physical properties were strongly correlated with each factor. The correlation coefficient r, expressing the strength of the relationship between the factor score x (mean value for 20 participants) and the physical properties y of each of the images, is We took the same approach with the properties of the music. As the impression value of the music we used the difference between the impression when the image was presented without music and that when it was presented with music. Finally, we tested the statistical significance of each of the calculated correlation coefficients. The statistical significance of r was tested with the t-test [22]. The test statistic t 0 is expressed by (3) (4) Table 5 lists the results of correlation analysis for each factor, and shows the physical properties which achieved significance at the 5% level or marginal significance in connection with the factors. The first factor, comfort, was correlated with direction feature, especially transverse lines and the saturation-entropy of the image, and the SD and maximum value of the power. The second factor, activity, was correlated with the saturation of the image, the mean power, and the low and high frequency power. Table 6 presents the correlations with each of the impression words. Tables 7 and 8 show the relation between the physical properties and impression words. From Table 6, we can see that saturation is correlated with the largest number of impression words. High-saturation images seem to have impressions such as warm, bright, soft, active, light, fancy. These words include many adjectives expressing potency and activity, which is consistent with the result of correlation analysis with factors. Furthermore, the impressions of images which contain high levels of green elements are bright, cheerful, and fancy. The images which contain high levels of blue elements have a wet impression. High-intensity images have a fancy impression. The green entropy is linked to wet dry. This means that images whose green element distribution is unimodal have Table 6. Results of correlation analysis (impression words) 58

7 Table 7. Relationship between impression words and physical properties of images a wet impression. Similarly, images whose saturation distribution is unimodal have a loose and extended impression. This is consistent with the result that the saturation-entropy is correlated to comfort in the correlation Table 8. Relationship between impression words and physical properties of music analysis of factors. Furthermore the hue-entropy is linked to wet dry. As regards the direction feature, images with many transverse lines have a soft impression. Next, we describe the correlation with the physical properties of music. Music with a high mean power gives impressions of activity, light, fancy, and clear to the images, and music with considerable variation in volume makes a warm and clear impression. Music with high mean values at low and high frequencies gives impressions of activity, light, fancy, and clear. Music with a high mean value at high frequency alters the pleasant and warm impression. We can see from the above that the power of the music is strongly correlated with adjectives expressing potency and activity. The frequency at the maximum power value is correlated with excited calm. To summarize, in the relationship between the image impression and the physical properties of the image, it has been shown that color information, especially the green element, saturation, and intensity are related to impression words expressing potency and activity. In addition, the saturation-entropy seems to be correlated with words representing spatial extent. However, there are some impression words with high factor loadings (e.g., pleasant unpleasant, beautiful ugly), although they are not linked to any impression words. Thus, it will be necessary to perform an analysis using physical properties which are not considered in this paper, such as the degree of circularity. In the relationship between the magnitude of changes in the image impression caused by music and the physical properties of music, the mean, SD, MLFP, and MHFP are correlated with some impression words expressing potency and activity. The max property is also related to impression words expressing potency and activity. Finally considering the results of changes in image impression based on Tables 7 and 8, the max value and frequency of music are high for image 1. Therefore, we believe that these features enhance potency and activity. On the other hand, one reason for the decreased levels of comfort is because adding music to the image with a high score for comfort and potency and activity makes the impression somewhat boisterous. Moreover, the music with image 4 gave a dark impression in the preliminary questionnaire. However, this music improves the score for comfort. The music with image 4 had the lowest frequency levels, but does not explain why the comfort score was improved. One reason might be that adding music to an image with a very low comfort score influences this result. Images 1 and 4 both had combinations of image and music that produced the same impression (bright bright, dark dark), and thus we predicted that consonance occurred. However, the music does not alter the impression in the same direction. It will be necessary to analyze this result in detail in the future. 59

8 5. Conclusions We investigated the effects of music on image impression, considering physical properties. As the first step, general scenery images as visual information and piano music as auditory information were used in experiments on image impressions when images were presented without music and with music, using the semantic differential method. Using data from this experiment, we examined the relationship between the changes in impression and the physical properties of each stimulus. We first used factor analysis to identify the factors expressing the image impression and the changes in the subjective impression when the image was presented without music and with music. In order to reveal the relationship between the impression of an image or music and physical properties, we extracted certain physical properties. As physical properties of the images, we considered the average color value and the entropy in the RGB value and HSI color spaces, and direction features obtained from a Gabor filter. Moreover, the mean power, standard deviation, and frequency were used as the physical properties of the music. We then calculated the correlation between the impression value and the magnitudes of the physical properties. The main findings were as follows. The color information of the image was strongly correlated with adjectives expressing potency and activity, and the entropy of saturation was correlated with words expressing spatial extent. The physical properties representing the power of the music were related to impression words expressing potency and activity. In the future we will conduct experiments using more stimuli and will focus on some impression words not linked to any physical properties. It will be necessary to make comparisons with other physical properties such as the degree of circularity of the images, and the tempo or rhythm of the music. Furthermore, by adding different music to one image, we will investigate the relationship between the changes in image impression and the physical properties of music in detail, and will apply the results to music selection methods in AV media. REFERENCES 1. Oyama T, Imai S, Wake T (editors). Handbook of psychology of sensation and perception, new edition. Seishin Shobo; (in Japanese) 2. Iwamiya S. Interaction between sound and picture when listening to music via audio-visual media. J IEICE 1995;78: (in Japanese) 3. Iwamiya S. The interaction between auditory and visual processing when listening to music via audiovisual media. J Acoust Soc Jpn 1992;43: (in Japanese) 4. Miyakawa M, Suzuki S, Aono S, Takagi K. The effects of visual information on the impressions of environmental sounds. J Acoust Soc Jpn 2000;56: (in Japanese) 5. Suzuki J, Sagawa Y, Sugie N. Dependency of subjective impression on the combinations of musics and images. J ITE 2001;55: (in Japanese) 6. Koga H, Shimoshio Y, Oyama Y. Selection method of music suitable for motion pictures with kansei information. Tech Rep IEICE, HCS, Vol. 99, No. 320, p 25 32, (in Japanese) 7. Iwamiya S, Hayashi K. The effects of color on the impression of music. Geijutsu Kogaku: J Design Kyushu Univ 1999;1: (in Japanese) 8. Bolivar VJ, Cohen AJ, Fentress JC. Semantic and formal congruency in music and motion pictures: Effects on the interpretation of visual action. Psychomusicology 1994;13: Iwamiya S. Audio-visual communication in visual media productions. Tech Rep IEICE, HIP, Vol. 102, No. 533, p 39 46, (in Japanese) 10. Iwamiya S, Jogetsu Y, Sugano Y, Takada M. Effects of musical tempo and tonality and visual speed and density on perception of audiovisual content. J Music Percept Cogn 2002;8: (in Japanese) 11. Sugano Y, Iwamiya S. The effects of the synchronization between auditory and visual accents and those of matching between musical tempo and visual speed on the emotional impression of combinations of motion picture and music. J Acoust Soc Jpn 2000;56: (in Japanese) 12. Kim K, Iwamiya S, Kitano H. Subjective congruence between rotating movement of a visual image and pitch modulation of a sound. Jpn J Ergon 2009;45: (in Japanese) 13. Su X, Kim K, Iwamiya S. Subjective congruence between switching patterns of moving pictures and pitch shift of sound. J Acoust Soc Jpn 2009;65: (in Japanese) 14. Healing & Relaxation Photo Image Database, Osgood CE, Suci GJ, Tennenbaum PH. The measurement of meaning. University of Illinois Press; Shibata T, Kato T. Image retrieval system for streetlandscape images using adjectives. J IEICE 1999;J82-D-I: (in Japanese) 17. Mukunoki M, Tanaka H, Ikeda K. An image retrieval system using the feature space consisting of antonymous pairs of adjectives. J Inf Process 2001;42: (in Japanese) 18. Sato K, Mitsukura Y. The effects of music on image impression. The relation between physical feature and Kansei information. Proc Japan Industry Appli- 60

9 cations Society Conference, p , (in Japanese) 19. The Color Science Association of Japan. Handbook of color science, new edition. University of Tokyo Press; (in Japanese) 20. Jain AK, Ratha NK, Lakshmanan S. Object detection using Gabor filters. Pattern Recognition 1997;30: Hayashi I, Williamson JR. A proposal of TAM network with Gabor type receptive field. J Jpn Soc Fuzzy Theory Intell Inf 2006;18: (in Japanese) 22. Mori T, Yoshida T. Technical book of data analysis for psychology. Kitaoji Shobo; (in Japanese) AUTHORS (from left to right) Keiko Sato received her B.S. and M.S. degrees from the Faculty of Education, Okayama University, in 2005 and 2007, and D.Eng. degree from the Graduate School of Bio-Applications and Systems Engineering, Tokyo University of Agriculture and Technology. Since 2008 she has been affiliated with the Department of Psychology, Ritsumeikan University, where she is currently a research associate. Her research interests include kansei information processing. Yasue Mitsukura received a doctoral degree from Tokushima University in She was a research associate at Tokushima University from 1999 to 2003, a lecturer at Okayama University from 2003 to 2006, and an associate professor at Tokyo University of Agriculture and Technology from 2006 to Since 2011 she has been an associate professor in the Department of System Design Engineering, Keio University. Her research interests include image processing, electroencephalogram analysis, and signal processing. 61