Musicolor: is there a link among mood, color and music?

Transcription

1 Musicolor: is there a link among mood, color and music? Andressa Eloisa Valengo Universidade Federal do Paraná Departamento de Informática aev11@inf.ufpr.br Francine Machado Resende Universidade Federal do Paraná Departamento de Informática fmr12@inf.ufpr.br Abstract In order to connect color, mood and music, the present work claims to search for the best way to classify musical and color data, relating them to emotions and mood. The database was created through an online form, linking three colors to each song. After that, a manual classification of the songs was made and the songs names were replaced for their respective mood. The next step is the classification and clustering tests. For this problem, some techniques implemented in the scikit-learn Python package were used. The classification was performed using k-neighbors Classifier and Support Vector Machine (SVM) while clustering was made using the K-means algorithm. Besides the use of different methods of classification, variations in datasets were also adopted to test different approaches. The best results were obtained for classification using the knn algorithm and binary datasets (only happy/sad). 1. Introduction It is almost an intuitive notion that music can evoke feelings in human beings. This notion has long been used for purposes as marketing [4] and therapeutic effects [2]. More recently, scientists have even explored the means by which this emotions are induced [8] and which aspects of music are related to the emotions [7]. Another curious notion is that we relate music to colors [14]. As colors are also associated with emotions [21, 13], we can hypothesize that the relations of music to colors are mediated by emotions. When the same piece of music was played with different emotional intentions, listeners attributed different color profiles to the piece [3]. In a recent study [15], this notion that color-music associations are mediated by emotion was investigated. In this study, participants first associated pieces of music with emotions, and chose colors that matched that music. Then, participants associated emotional faces to the colors and to the music. Results show that indeed musiccolor associations seem to be mediated by emotions [15]. Research in this field seem to point to the fact that faster or major-key music is associated with brighter, yellower colors [3, 10, 15, 16, 20]. Recently, Lindborg and Friberg [10] designed a beautiful study to further analyze color-music associations. After asking for the listeners to attribute a color to some musics, they tried to use the data to predict the color tat would be associated with the music, by multiple regression analysis. The prediction was successful for the Lightness parameter of the color. Additionaly, the model used for the prediction was more successful when it included the emotions attributed to the music in addition to acoustic parameters [10]. From this curious connection between color and music, came an idea to make a project that aims to use color, as a reflection of the person s mood, to predict the music the person wants to listen. The main objective of the present work was to search for the best way to classify musical and color data, relating them to emotions and mood, as well as to make sure if they are consistent and enough to start the project Related Works Attempts to automatically classify the emotion of music are being made. The main difficulties are on which audio features and which mood categories to use. Using BP neural networks, [5] classified music in four mood categories with a precision of 67%. A Support Vector Machine (SVM) active learning method was also used, achieving only a 50% accuracy on mood classification [12]. Lu et al. [11] implemented an hierarquical framework for mood detection and achieved an 86% accuracy. Using K-means clusterization, Hu et al. [6] established a ground-truth set for Music Information Retrieval (MIR) systems. The 3 established clusters seem to match categories classified emotionally as aggressive/angry, mellow/calm, or upbeat/happy [6]. To check if style and mood tags could be propagated, Sordo et al. [19] used content-based (CB) similarity and propagation was successful for happy, angry, and sad tags, but not to Mysterious tags. Skowronek et al. [17] classified music in 12 mood categories, and the performance var- 1

2 ied from 77% to 91%, depending on the category. Different kinds of audio features to be used for SVM mood classification showed that spectral features performed better than rhythm or dinamics features [18]. An attempt to reduce the audio features using both MRMR and PCA methods with SVM classifiers, showed that only 27,9% of 538 features were really contributing to the classification [1]. Adding the lyrics to the analyses seemed to optimize music classification for some emotions, specially for happy and sad categories [9]. A mood classification based only on the lyrics was also tried, achieving a 60% accuracy [23]. An important cultural issue was also analyzed, verifying if audio features, mood categories, and classification models developed for Western music were also applicable for non- Western, more specifically Chinese, music. The mood categories were applicable, and the same features could be used, however, when using English music as training to classify Chinese music, or vice-versa, the performance was not as good [22]. To the best of our knowledge, no studies included Brazilian music, as the present study does (in addition to North American and European music). Most importantly, this is the first attempt to use colors as a way of classifying music trough a machine learning method. 2. Methods 2.1. Dataset Creation To establish the database, a survey was created and shared through social networks. The survey included a question for the name of the person, a question with 24 color options (Figure 1) from which the person had to choose three (that suited him/her best at that moment), and a question for the person to write the song (artist - song name) he/she would like to listen at that moment. The color options go from bright shades to dark ones and they were shuffled to split similar colors or shades. The survey had a total of 654 answers. In the dataset, the input was not the song itself but a classification in which the song was fitted. To classify the songs, answers were first organized alphabetically according to the name of the artist and the song, to ensure that the same song always received the same classification. Repeated answers (n = 5), with the same person, same song and same colors, with a short period of time between the answers, where excluded. Incomplete or incorrect answers (songs that do not exist) where also excluded. A total of 642 answers remained for classification. The songs were listened to in YouTube, at least until the first chorus, always preferring the original audio, not live performances, and not looking at the video. To the classification, the rhythm of the song was considered and the lyrics were only taken into consideration when their Table 1. Number (N) of answers for each category Category N Category N angry 15 relax 19 betrayed 6 religious 1 childish 8 romantic excited 38 goodbye 4 romanic calm 68 happy 73 sad 75 hope 26 sexy 8 lady-killer 2 spooky 12 missing someone 15 thankful 5 needy 6 thoughtful excited 50 party 32 thoughtful calm 56 politicized 4 very happy 6 reborn 24 very sad 60 regret 26 victory 3 meaning caught the attention of the listener. The categories were: angry, betrayed, childish, goodbye, happy, hope, lady-killer, missing someone, needy, party, politicized, reborn, regret, relax, religious, romantic, sad, sexy, spooky, thankful, thoughtful, very happy, very sad, and victory. As there were too many songs in some categories, sugcategories were created to be included in the tests: calm romantic, excited romantic, calm thoughtful, and excited thoughtful Dataset Representation The complete dataset has 642 instances and each one of them has 3 colors and the label (category) membership. The Red Green Blue (RGB) system was selected to represent each color, this way each instance has nine attributes, because the colors in the RGB system are represented by 3 values ranging from 0 to 255. In addition, other three attributes were created for each instance, consisting of the mean of the three red values, of the three blue and of the three green values. Tests were performed using (a) the 9 values for the 3 colors, (b) the 3 average (AVG) values Algorithms and Techniques Once the dataset was created, the next task was to perform the color-to-category classification. In this step, machine learning was applied by using classification and clustering techniques implemented in the scikit-learn Python package. Even though this problem clearly involved supervised learning, clustering was applied in order to evaluate the music-to-category classification and also to verify how close to each other were the categories. Supervised learning was performed using k-neighbors Classifier and Support Vector Machine (SVM), whilst K-means was used for clustering. Different algorithms were used for classification to find out which one handles the best this kind of problem. 2

3 Figure 1. Colors included on the survey Tests Figure 2 represents the pipeline for the tests. Different input datasets for each representation were used for each classification or clustering method and their parameter variations. Firstly, tests were done using all instances (N = 642) with and without (default) subcategories from romantic and thoughtful categories. Because some feelings are easier to recognize than others, other dataset configurations were created: only happy or sad (N = 214), only positive or negative categories (N = 532), and strict positive or negative categories (N = 364). The configuration only happy or sad includes only the instances classified as happy, very happy, sad and very sad. The configuration only positive or negative represents the categories classified as negative (sad, very sad, needy, missing someone, regret, spooky, angry, betrayed, goodbye) or positive (happy, very happy, reborn, victory, thankful, hope, relax, religious, sexy, lady-killer, party, romantic). The configuration strict positive or negative is an intermediate between only happy or sad and only negative or positive, including on positive the categories happy, very happy, reborn, victory, thankful, and party; and on negative sad, very sad, needy, missing someone, regret, spooky, angry, betrayed, and goodbye. Each configuration was split in two parts of 90% training and 10% test by using the train test split function from scikit-learn. In the second round of tests using the k- Neighbors Classifiers, different numbers of neighbors were used for each dataset configuration, according to the respective performances in the first round: only happy or sad = [3, 5, 7, 9, 11, 13, 15], only positive or negative = [10, 20, 30, 40, 50, 60, 70], and strict positive or negative = [20, 25, 30, 35, 40]. In addition, these tests included only the AVG representation. 3. Results 3.1. Classification When testing weights parameters of k-neighbors Classifiers, uniform parameter always performed better than distance parameter (Table 2), probably because the instances on the training sets are close to each other in the space, so it is better that all neighbors have the same weight when voting the category for the new instance. Considering the three color representation (all features) and the average (AVG), in the vast majority of cases, tests with the AVG representation were more accurate (Table 2), because AVG represents a mix of the three colors, what is reasonable when considering that the colors are supposed to stand for moods. The main idea is that, for a given mood, one would choose colors that are in a way close to each other (brightness, shade, and saturation). The chosen colors also may have similar RGB values. Tests with restrictive configurations (only happy or sad, only positive or negative, and strict positive or negative) showed more accurate results (Table 2) than the default datasets (with and without subcategories) because it is easier to assign categories, to the musics, when the moods are 3

4 Figure 2. Pipeline of the tests. more contrasting. This reflects the difficulty encountered when making music-to-category classification. Even though the results for the binary configurations were more accurate (Table 2), it must be said that if random assignment was made, a 50% accuracy was expected, thus, the accuracy have to be interpreted considering 50% as baseline. Hence, the results obtained using the default datasets may not be that bad. The three k-neighbors classifiers (knn and its variants) had almost the same accuracy for different numbers of neighbors (Figure 3). In addition, for the only happy or sad configuration, classification using five neighbors was more accurate (Figure 3 - A). On the other hand, only positive or negative and strict positive or negative better accuracy values were achieved using eighty-five (Figure 3 - B) and twenty-five neighbors (Figure 3 - C), respectively. Considering the results obtained in the second round of tests (Figure 3), cross-validation was performed in order to Table 2. Accuracy of k-neighbors Classifier tests. The accuracy is the value of the highest peak obtained in the tests among the k-nearest Neighbor (KNN) algorithm and its two variants kdtree and BallTree. ALL: All Features; AVG: Average. distance uniform ALL AVG ALL AVG Default 18% 20% 20% 24% Default + Subcategories 14% 10% 16% 14% Only Happy/Sad 65% 70% 65% 75% Only Positive/Negative 50% 42% 54% 55% Strict Positive/Negative 46% 54% 55% 65% evaluate the classifier (knn - brute variation). As shown in Table 3, the average accuracy when classifying only happy or sad configuration was 64% (+/- 24%), while the configurations only positive or negative and strict positive or negative had mean accuracy values of 57% (+/-16%) and 4

5 Only Happy/Sad Only +/- Strict +/- 50% 42.59% 40.54% Table 4. SVM accuracy for the average color representation Only Happy/Sad Only +/- Strict +/ % 40.74% 43.24% Table 5. SVM accuracy for the three color representation ues (Table 4) than the three color representation (Table 5) for two of the configurations. These SVM results (Table 4, Table 5 ) show that k- Neighbors classifiers performed better than SVM on our dataset. On the other hand, in the literature, music mood classification is best achieved using SVM. This difference is probably because in our dataset the moods are represented in a different way, by colors. Figure 3. Accuracy for different number of Neighbors. The configurations only happy or sad, only positive or negative, and strict positive or negative were used in A, B, and C respectively. Table 3. Cross-validation for KNN. The cross-validation was performed on the training data (AVG representation) using ten folds. In the second, third and fourth columns are shown the accuracy values for only happy sad, only positive or negative, and strict positive or negative configurations. Fold Happy/Sad Only +/- Strict +/- 1 st 45.0% 73.5% 60.6% 2 nd 65.0% 61.2% 57.7% 3 rd 60.0% 59.2% 63.6% 4 th 75.0% 55.1% 60.6% 5 th 85.0% 53.1% 60.6% 6 th 63.2% 55.3% 60.6% 7 th 52.6% 40.4% 48.5% 8 th 77.7% 51.1% 51.5% 9 th % 62.5% 10 th 50.0% 58.7% 61.3% Mean 64 (+/-24)% 57 (+/-16)% 59 (+/-9)% 59% (+/-9%), respectively. Confirming that only happy and sad is the best configuration. When testing the representations of the dataset using the SVM classifier (average and three color), once again, the average (AVG) representation showed better accuracy val Clustering In order to evaluate the music-to-category classification, clustering was performed. The results (Image 4, Image 5) show that for all representation configurations, the higher the number of clusters, the best is the silhouette value. This improvement is probably not relevant, because it may reflect that the clusters are overlapping, since all values are close to zero. When considering two clusters, some categories were more assigned to one cluster than the other. The percentage of each category that was placed in each cluster is showed in Table 6. The categories sad, very sad, and angry were mostly assigned to the same cluster (C1). Whilst happy, party, and reborn were mostly assigned to the remaining cluster (C2). This supports the idea that binary classifications (as in happy/sad configuration) are better than complex classifications. 4. Conclusion and Future Work To the best of our knowledge, this is the first study to relate color-mood-music applying machine learning. For classification, k-neighbors and SVM were tested, and K- means was used for clustering. Among knn and its implementation variations (KdTree and BallTree), knn was more accurate. In addition, knn showed better results than SVM. Different datasets were created, and using the binary ones for classification, better accuracy values were obtained. This is probably because it is easier to classify music moods when they are contrasting, as was the case of only happy and sad configuration. One limitation of the present study was the small number of instances in the datasets. The results would probably be better if the datasets were larger. Additionally, the musiccategory could be improved if at least three people classified 5

6 Figure 4. Silhouette analysis for kmeans clustering on average representation. Table 6. Statistics summary of the 2 clusters of default dataset C1 C2 Category 43.5% 56.5% happy 60.9% 39.1% regret 42.3% 57.7% party 44.4% 55.6% calm 36.4% 63.6% hope 58.2% 41.8% very sad 90.0% 10.0% spooky 64.7% 35.3% sad 67.4% 32.6% thoughtful excited 55.1% 44.9% thoughtful calm 71.4% 28.6% angry 40.0% 60.0% romantic excited 37.3% 62.7% romantic calm 46.2% 53.8% missing someone 36.4% 63.6% reborn Figure 5. Silhouette analysis for kmeans clustering on three colors representation. the data, or even if an automatic classification method was used. In further works the ground truth mood classification of music can be obtained from existing datasets, for example from last.fm API. But one drawback would be the lack of Brazilian music in those databases. In case of the creation of a new dataset (color-mood-music), it would be interesting to develop a system where a person could listen to a music and attribute colors to that music. This work was a first step to relate color and music through mood and it shows that the idea is feasible and worthy of further studies. References [1] B. K. Baniya and C. S. Hong. Music Mood Classification using Reduced Audio Features., pages , [2] C. F. Barber. The use of music and colour theory as a behaviour modifier. British journal of nursing (Mark Allen Publishing), 8(7):443 8, [3] R. Bresin. What is the color of that music performance? Proceedings of the International Computer Music Conference, pages , [4] G. C. Bruner II. Music, Mood, and Marketing. Journal of Marketing, 54(4):94 104, [5] Y. Feng, Y. Zhuang, and Y. Pan. Music information retrieval by detecting mood via computational media aesthetics. Proceedings IEEE/WIC International Conference on Web Intelligence (WI 2003), pages , [6] X. Hu, M. Bay, and J. Downie. Creating a Simplified Music Mood Classification Ground-Truth Set. ISMIR, [7] P. G. Hunter, E. G. Schellenberg, and U. Schimmack. Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1):47 56, [8] P. N. Justin and D. Västfjäll. Emotional Responses to Music: The Need to Consider Underlying Mechanisms. Behavioral and Brain Sciences, 31(5): , [9] C. Laurier, J. Grivolla, and P. Herrera. Multimodal Music Mood Classification Using Audio and Lyrics. Machine Learning and Applications, ICMLA 08., pages , [10] P. Lindborg and A. K. Friberg. Colour Association with Music Is Mediated by Emotion: Evidence from an Experiment Using a CIE Lab Interface and Interviews. PloS one, 10(12):e , [11] L. Lu, D. Liu, and H. J. Zhang. Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech and Language Processing, 14(1):5 18, [12] M. I. Mandel, G. E. Poliner, and D. P. W. Ellis. Support vector machine active learning for music retrieval. Multimedia Systems, 12(1):3 13,

7 [13] K. Naz and H. Epps. Relationship between color and emotion: a study of college students. College Student J, 38(3): , [14] H. S. Odbert, T. F. Karwoski, and a. B. Eckerson. Studies In Synesthetic Thinking: I. Musical and Verbal Associations of Color and Mood. The Journal of General Psychology, 26: , [15] S. E. Palmer, K. B. Schloss, Z. Xu, and L. R. Prado-Leon. Music-color associations are mediated by emotion. Proceedings of the National Academy of Sciences, 110(22): , [16] D. J. POLZELLA and J. L. HASSEN. Aesthetic preferences for combinations of color and music. Perceptual and Motor Skills, 85(3): , Dec [17] J. Skowronek, M. F. McKinney, and S. Van De Par. A Demonstrator for Automatic Music Mood Estimation. IS- MIR, pages , [18] Y. Song, S. Dixon, and M. Pearce. Evaluation of Musical Features for Emotion Classification. International Society for Music Information Retrieval Conference (ISMIR), pages , [19] M. Sordo, C. Laurier, and Ò. Celma. Annotating Music Collections : How Content-Based Similarity Helps to Propagate Labels. ISMIR, pages , [20] T. Tsang and K. B. Schloss. Associations between Color and Music are Mediated by Emotion and Influenced by Tempo. the Yale Review of Undergraduate Research in Psychology, pages 82 93, [21] L. B. Wexner. The Degree to Which Colors (Hues) Are Associated with Mood-Tones. The Journal of Applied Psychology, 38(6): , [22] Y.-H. Yang and X. Hu. Cross-cultural music mood classification: A comparison on English and Chinese songs. 13th International Society for Music Information Retrieval Conference (ISMIR), pages 19 24, [23] T. C. Ying, S. Doraisamy, and L. N. Abdullah. Genre and mood classification using lyric features International Conference on Information Retrieval & Knowledge Management (CAMP), pages ,