Toward Web 2.0 music information retrieval: Utilizing emotion-based, user-assigned descriptors

Transcription

1 Toward Web 2.0 music information retrieval: Utilizing emotion-based, user-assigned descriptors Hyuk-Jin Lee School of Library and Information Studies, Texas Woman's University, Stoddard Hall, Room 414, PO Box , Denton, TX Diane Neal School of Library and Information Sciences, North Carolina Central University, James E. Shepard Memorial Library, 1801 Fayetteville St., Durham, NC This exploratory, preliminary study seeks to determine whether the use of emotion-based, user-assigned descriptors elicited from music information retrieval (MIR) system users could improve access and retrieval for a browsing-based MIR system within the Web 2.0 environment. We used the Glass Engine, a Web-based tool designed for exploring Philip Glass' music, as the basis for conceptualizing this study. The Glass Engine allows users to browse several facets of Glass' music via direct manipulation (sliding bars). Introduction and previous studies Emotion is one of the core factors in Music Information Retrieval (MIR) as recognized by several researchers (Chai & Vercoe, 2000; Kim & Belkin, 2002). Woody and Burns (2001) discuss the theory of mood management, which argues that people choose entertainment to direct their mood in a desired direction. However, few or no attempts have been conducted to apply emotional factors to MIR systems in the Web 2.0 environment. The metadata about the multimedia content itself are still insufficient, because many features of multimedia content are quite perceptual and user-dependent (Chai & Vercoe, 2000). Chai and Vercoe (2000) indicated that emotional features are very important for multimedia retrieval, but it is difficult to describe them using a universal model since different users may have varying emotional responses to the same multimedia content. Huron (2000) said that while the basic information such as the name of the composer or the title of the work remains important in MIR, these standard reference tags have very limited applicability in most music-related queries. He also added that although there are many other uses for music, music s preeminent functions are social and psychological, and typically, the most useful retrieval indexes focus on stylistic, mood, and similarity information. Finally, the author indicated a very important point: the textual information

2 retrieval on the Internet shows that non-proprietary decentralized indexing is likely to prove more popular and successful. Similarly, future music indexes will likely resemble Web-wide search engines rather than closed proprietary systems. Rolland (2001) also emphasized that the content-based MIR user model should be adaptive. That is, instead of using one general user model, the MIR system must adjust the similarity model based on user feedback received during successive interactions with the user. Research Framework Based on the necessities mentioned above, our study investigates how to embody an MIR system that reflects unique user-assigned, emotion-based music metadata, which is different from traditional bibliographic description, in the Web 2.0 environment. Glass Engine Faceted, emotion-based description and access is the focus of this study. We plan to develop a graphical user interface similar to the one utilized in the Glass Engine (available at ; also see Figure 1), which uses sliding bars similar to Alphasliders (Ahlberg & Shneiderman, 1994) that allow the user to browse Philip Glass compositions and listen to short clips of his music. The interface includes the sliding bars, which represent various facets of the music objects in the system s database. We believe the bar interface of this system might be effective for browsing in MIR systems. However, as Neal (2004) indicated, only system developers know the structure of the database that stores the music and its metadata, a factor that hinders creating effective representation for object access. In addition, Glass producer was the only person who decided how the Glass Engine's various facets would apply to the music, including the emotional facets of joy and sorrow (Podlaseck, 2003). These factors provide a limited basis for accurate, meaningful description, and do not reflect any user feedback.

3 Figure 1. Glass Engine Research questions This study investigates how the sliding bar interface such as the one used in the Glass Engine can be modified and used more effectively by MIR system users. First, we would like to see the patterns of the sliding bar usage from 1) selected Philip Glass music, which could be considered modern music with classical music influences, and 2) works by other composers from the classical music genre. Second, we will apply the Web 2.0 concept of tagging as we believe that the users' collective knowledge, represented by the collected patterns of the emotional responses using the sliding bars, will meet the necessity of an effective and adaptive browsing-based MIR system. Therefore, there are three research questions drawn from the general problem statement and previous studies. Research Question 1: This study aims to find out whether there are some distinguished patterns in five different basic emotional facets overall. Research Question 2: This study aims to find out whether there is any difference in the users reactions between classical music and Glass' music. Research Question 3: Based on the findings from the first two questions, this study aims to find out whether the collected patterns of the emotional responses will meet the necessity of an effective and adaptive browsing-based MIR system.

4 Methodology A total of 101 individuals from seven different fields (mostly library and information studies and music) at Texas Woman s University participated in this preliminary study. This resulted in 65 usable responses, mainly due to participants failing to complete the entire survey. We used SurveyMonkey.com to collect data. Once participants started the survey, they indicated their field of study. The participants listened to one-minute segments from five Philip Glass music pieces and five classical music pieces written by other composers. We ordered the music pieces within the survey to avoid any side effects from listening to the same composers' music continuously. In addition, we randomly offered the five emotions. The order of the total 10 music pieces as we provided them to the participants is in Table 1. Order Emotional facet Genre Title (composer) 1 Happiness Classical Four Seasons, Spring movement (Vivaldi) 2 Anger Glass Akhnaten, Attack movement (Glass) 3 Sadness Classical Piano Concerto No. 2 (Rachmaninoff) 4 Disgust Glass Akhnaten, Temple movement (Glass) 5 Fear Classical Symphony No. 9 (Shostakovich) 6 Sadness Glass Glassworks, Opening movement (Glass) 7 Anger Classical The Planets, Mars movement (Holst) 8 Happiness Glass Einstein on the Beach, Spaceship movement (Glass) 9 Disgust Classical Symphony No. 14 (Shostakovich) 10 Fear Glass Glassworks, Floe movement (Glass) Table 1. The order of the 10 music pieces We chose the music pieces based on our personal judgments of the primary emotion elicited by the music (Lee & Neal, 2007). After listening to each piece, participants indicated the amount of each of five emotions the music clip caused them to feel. We regard each emotion as analogous to the sliding bars present in the Glass Engine. The emotions presented to participants include the five emotions from Power s (2006) Basic Emotions Scale: Happiness (joy, cheerful, loving), Sadness (despair, miserable, gloomy, mournful), Anger (frustration, irritation, aggression), Fear (anxious, nervousness, tension, worry), and Disgust (shame, guilt, humiliated, blameworthy). In this preliminary study, each sliding bar is represented by and composed of nine Likert scale ratings (Not at all = 1, Not very much = 2-3, Neutral = 4-5, Somewhat = 6-7, and Very much = 8-9). The higher score a participant indicates, the stronger emotion a specific music piece elicited for that participant (see Figure 2.) Therefore, a total of 650 sliding bar patterns were collected, where one pattern comprises five emotional categories.

5 Figure 2. Emotional facet Likert scale ratings Data Analysis We analyzed whether patterns emerged from the participants collective responses (Suriowecki, 2004). Correlation among emotional reactions from the sample may help deduce whether music evokes similar responses among individuals. As part of our ongoing MIR study, we would like to investigate whether the collected bar patterns match the music users searching needs. The collected bar patterns, which show the subjects emotional reactions to the music, will be considered as the tags. Participants who need to find a certain music piece that matches their desired emotion-based music style could search the music by simply clicking on the tag of the appropriate bar pattern. Then, we may see if the collected emotional patterns will be easy to use as well as meet music listeners' emotional information needs in the Web 2.0 environment. Preliminary Findings General In this study, we have found some interesting overall patterns in participants' reactions to the music. First, with the exception of Happiness for the "Spring" movement from Vivaldi's "Four Seasons", which received an average rating of 8.2 out of a possible 9, they did not rank emotions very highly (see Appendix A). The average rank of the most highly rated emotion for each piece was 6.2 out of 9. Vivaldi's "Four Seasons" was the only piece that elicited a clearly favored emotion (Happiness). The other pieces resulted in either bimodal favorites or somewhat even distributions (see also Appendix A). For example, Shostakovich's "Symphony No. 9", which

6 we used to represent Fear, resulted in a ranking of 5.5 for Sadness and 5.9 for Fear. Fear was the most frequently chosen emotion among the participants; they rated it the most highly for six of the 10 pieces on average. They favored Happiness for two pieces, and sadness for two pieces. Their most frequently chosen emotion agreed with our perception of the predominant emotion in five of the 10 pieces. Correlations among the emotional facets We tested correlations among the emotions as well. Correlation coefficients were computed among the five emotional facets to find out whether there were any interesting relationships among one another. All the statistically significant results of the correlational analyses per each measurement are presented in Appendices B, C, and D. This section only covers the major findings from the correlational analyses. From all data (N = 650), we found that Happiness is negatively correlated with the other four emotions. However, the correlations of Sadness showed that r was significantly positive with Disgust and Fear. The correlations of Anger showed that r was significantly positive with Fear and Disgust. Finally, Fear was significantly correlated with Anger, Disgust, and Sadness (see Appendix C). Additionally, we found that the correlations in the classical music (N = 325) were more apparent than the ones in the Glass music (N = 325) (see Appendix D). There are some interesting correlational relationships in each music piece (total 10, N = 65) (see Appendix B). Here are the findings in order according to the given music pieces, which were selected to represent each emotional facet. Happiness in classical music (Q1) has weak negative correlations with the other emotions. Anger in Glass music (Q2) has a significant negative correlation with Happiness and orderly positive correlations with Fear, Sadness, and Disgust Sadness in classical music (Q3) has a significant negative correlation with Happiness and orderly positive correlations with Fear and Disgust. Disgust in Glass music (Q4) has a significant negative correlation with Happiness and orderly positive correlations with Sadness, Fear and Anger Fear in classical music (Q5) has a significant negative correlation with Happiness and orderly positive correlations with Anger and Disgust. Sadness in Glass music (Q6) has a significant negative correlation with Happiness and orderly positive correlations with Fear and Anger. Anger in classical music (Q7) has a significant negative correlation with Happiness and positive correlations with Sadness. Happiness in Glass music (Q8) has a significant negative correlation with Anger. Disgust in classical music (Q9) has a significant positive correlation with Sadness. Fear in Glass music (Q10) has a significant negative correlation with Happiness and orderly positive correlations with Anger and Disgust. These findings are arranged in Table 2. As you can see, each item in the cells represents the

7 given music piece. For example, S (C) is the classical music piece representing Sadness. Plus or minus indicates whether that music piece has a positive or negative correlation between the two emotions. HAPPINESS SADNESS ANGER FEAR DISGUST HAPPINESS - S (C) - A (C) - F (C) SADNESS - S (G) + A (C) + S (C) + D (C)+ S (C) ANGER - H (G) + A (G)+ S (G) + F (C) FEAR - F (G) + S (G) + A (G)+ F (G) + F (C) DISGUST - D (G) + D (G) + A(G)+ D (G) + D (G)+ F (G) Table 2. Correlations among 5 emotions in classical and Glass music Dark green colored cells: Classical music pieces Orange colored cells: Philip Glass music pieces H: Happiness; A: Anger, S: Sadness, F: Fear, D: Disgust C: Classical music, G: Philip Glass music Influences from different musical backgrounds Finally, we found that different musical backgrounds may influence the patterns of reactions to the basic emotional facets. For example, as you can see in the bar graphs in Appendix E, music majors tended to give higher emotional rankings than library and information studies majors for Glass' music than for classical music. The average of all emotional ratings for classical music for music majors was 3.6, while it was 4.1 for Glass' music. The average of library and information studies' studies emotional ratings was 3.5 for classical music and 3.8 for Glass' music. This may tell us that music majors could determine the "correct" emotions, especially for Glass' compositions, better than library and information studies students. Also, some patterns showed exceptional differences between the two groups of the subjects, such as Anger and Fear in classical music, as well as Sadness and Disgust in Glass' music. We plan to investigate whether these phenomena occurred by chance in further detail. Conclusions and Future Study Based on the results of this preliminary study, it seems that assigning emotions to works of classical and Philip Glass music for browsing and searching is best left to the users of an MIR system. Only one work demonstrated a clearly favored emotion among participants, and the other works showed more evenly distributed emotional responses. Given these results, it does not seem as useful to employ emotional rankings assigned by only one person (as in the case of the Glass Engine) as it seems to gather emotional rankings from all interested users. This study may contribute to a social computing approach to MIR; specifically,

8 emotion-based faceted access to music. The findings in this study may be applicable in developing an MIR system combining both the emotion-based sliding bars, which reflects user-assigned emotional facets, and a keyword search. Using this system, a user could locate a Cheerful but calm work by Franz Peter Schubert. More findings about the relationships among the emotional facets may contribute to suggesting more optimized sliding bar patterns to the users. Finally, the findings of this study may be extended to other musical genres such as pop, rock, or jazz. For example, investigating the relationships among the emotional facets of different musical genres could result in interesting findings compared with the findings from this study. We plan to build an MIR system based on our findings and the Glass Engine that employs user-assigned emotional rankings and sliding bars for browsing. Then, we will observe how the new MIR system improves MIR users browsing and searching results. For future studies, we would like to compare how different music backgrounds influence emotional responses and searching results. Related to this topic, we also plan to investigate how results differ according to the participants' field of study such as library and information studies, music, and reading. References Ahlberg, C., & Shneiderman, B. (1994). The Alphaslider: A compact and rapid selector. Proceedings of ACM CHI 94 Conference on Human Factors in Information Systems, Chai, W., & Vercoe, B. (2000). Using user models in music information retrieval systems. Proceedings of the 1st Annual International Symposium on Music 330 Annual Review of Information Science and Technology Information Retrieval (ISMIR 2000). Huron, D. (2000). Perceptual and cognitive applications in music information retrieval. Proceedings of the 1st Annual International Symposium on Music Information Retrieval (ISMIR 2000). Kim, J.-Y., & Belkin, N. J. (2002). Categories of music description and search terms and phrases used by non-music experts. In Third International Conference on Music Information Retrieval (ISMIR 2002) (pp ). Lee, H.-J., & Neal, D. (2007). Image entry points of different genres of photographs for the Web searching environment. Annual Conference of the Association for Library and Information Science Education, Seattle, WA. Neal, D. (2004). The Glass Engine and direct manipulation. Unpublished manuscript. Podlaseck, M., Hoch, R., & Schoenberg, E. (2003). This is not a catalog: Philip Glass s works online. Multimedia IEEE, 10(4), 4-7. Power, M. J. (2006). The structure of emotion: An empirical comparison of six models.

9 Cognition & Emotion, 20 (5), Rolland, P.-Y., (2001). Adaptive user-modeling in a content-based music retrieval system. Proceedings of the 2nd Annual International Symposium on Music Information Retrieval (ISMIR 2001). Suriowecki, J. (2004). The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Doubleday. Woody, R.H., & Burns, K. J. (2001). Predicting music appreciation with past emotional responses to music. Journal of Research in Music Education, 49(1), Appendix A. Five emotions per classical and Glass music pieces Figure 3. Happiness (Classical)

10 Figure 4. Happiness (Glass)

11 Figure 5. Anger (Classical)

12 Figure 6. Anger (Glass)

13 Figure 7. Sadness (Classical)

14 Figure 8. Sadness (Glass)

15 Figure 9. Disgust (Classical)

16 Figure 10. Disgust (Glass)

17 Figure 11. Fear (Classical)

18 Figure 12. Fear (Glass) Appendix B. Correlations of 5 emotions per classic and Glass music pieces Table 3. Correlations (Q1: Happiness - Classical) Table 4. Correlations (Q2: Anger - Glass)

19 Table 5. Correlations (Q3: Sadness - Classical) Table 6. Correlations (Q4: Disgust - Glass)

20 Table 7. Correlations (Q5: Fear - Classical) Table 8. Correlations (Q6: Sadness - Glass)

21 Table 9. Correlations (Q7: Anger - Classical) Table 10. Correlations (Q8: Happiness - Glass)

22 Table 11. Correlations (Q9: Disgust - Classical) Table 12. Correlations (Q10: Fear - Glass)

23 Appendix C. Correlations (Total dataset) Table 13. Correlations (total) Appendix D. Correlations per classical and Glass music pieces Table 14. Correlations (Classical)

24 Table 15. Correlations (Glass) Appendix E. Five emotions per classical and Glass music pieces in LIS and Music majors

25 Figure 13. Happiness (Classical)

26 Figure 14. Happiness (Glass)

27 Figure 15. Anger (Classical)

28 Figure 16. Anger (Glass) Figure 17. Sadness (Classical)

29 Figure 18. Sadness (Glass)

30 Figure 19. Disgust (Classical) Figure 20. Disgust (Glass)

31 Figure 21. Fear (Classical)

32 Figure 22. Fear (Glass)