Overcoming Accuracy-Diversity Tradeoff in Recommender Systems: A Variance-Based Approach

Transcription

1 Overcoming Accuracy-Diversity Tradeoff in Recommender Systems: A Variance-Based Approach Gediminas Adomavicius gedas@umn.edu YoungOk Kwon ykwon@csom.umn.edu Department of Information and Decision Sciences Carlson School of Management, University of Minnesota Abstract. Collaborative filtering and, more generally, recommender systems represent an increasingly popular and important set of personalization technologies that help people navigate through the vast amounts of information. The performance of recommender systems can be evaluated along several dimensions, such as the accuracy of recommendations for each user and the diversity of recommendations across different users. Intuitively, there is a tradeoff between accuracy and diversity, because high accuracy may often be obtained by safely recommending to users the most popular ( bestselling ) items, which can lead to the reduction in recommendation diversity, i.e., less personalized recommendations. And conversely, higher diversity can be achieved by trying to uncover and recommend highly idiosyncratic/personalized items for each user, which are inherently more difficult to predict and, thus, may lead to a decrease in recommendation accuracy. In our research we explore different ways to overcome this accuracy-diversity tradeoff, and in this paper we discuss a variance-based approach that can improve both the accuracy and diversity of recommendations obtained from a traditional collaborative filtering technique. We provide empirical results based on several real-world movie rating datasets. Résumé. L'exécution des systèmes de recommandation peut être évaluée selon plusieurs dimensions, telles que l'exactitude prédictive des recommandations pour chaque utilisateur aussi bien que la diversité des recommandations parmi des utilisateurs différents. Dans notre recherche nous présentons une approche dispersion-basée qui peut améliorer l'exactitude et la diversité des recommandations obtenues à partir d'une technique traditionnelle de filtrage collaboratif. Nous fournissons des résultats empiriques basés sur plusieurs ensembles de données réels de cotes de films.. Introduction and Motivation Collaborative filtering (CF) and, more generally, recommender systems represent an increasingly popular and important set of personalization technologies that help people navigate through the vast amounts of information [0]. Recommender systems typically try to estimate the ratings of unknown items (or products) for each user, often based on other users ratings, and recommend the items with the highest predicted ratings. There have been many studies on developing new algorithms that can improve the accuracy of recommendations. However, relying on the accuracy of recommendations alone may not be enough to find the most relevant items for a user. The recommender systems should also be able to provide the user with highly idiosyncratic or personalized items, often measured by the diversity metric [2, 7]. Intuitively, there is a tradeoff between accuracy and diversity, because high accuracy may often be obtained by safely recommending to users the most popular items, which can lead to the reduction in diversity, i.e., less personalized recommendations. And conversely, higher diversity can be achieved by trying to uncover and recommend highly idiosyncratic or personalized items for each user, which are inherently more difficult to predict due to the lack of data, and, thus, may lead to a decrease in recommendation accuracy. The e-commerce research literature is not unanimous on the impact of recommender systems on sales diversity. On one hand, the long tail phenomenon that refers to the increase in the tail of the sales distribution (i.e., the increase in sales diversity) has been empirically confirmed using data from online clothing retailer [5]. On the other hand, a recent study [6] reports a contradictory finding that the use of recommender systems, including common CF techniques, can actually reduce the aggregate sales diversity. This can be explained by the fact that the items with limited historical data are more difficult to recommend to users, while popular items typically have more ratings and, therefore, can be recommended to more users. Clearly, the diversity of recommendations represents an important research topic that

2 needs further study. In this paper, we adopt a variance-based approach to improving the accuracy and diversity of recommendations using a traditional CF technique, and empirically demonstrate how this new approach can overcome the accuracy-diversity tradeoff. 2. Related Work Among the most popular and well-performing recommendation techniques are neighborhood-based CF techniques, which recommend to a particular user those items that other users with similar preferences (i.e., neighbors ) liked in the past [, 4]. There are multiple variations of such techniques (e.g., userbased vs. item-based CF []). One of the most popular approaches involves: (a) the calculation of the similarity between two users using a cosine similarity metric, and (b) the prediction of the rating of an unknown item for the particular user as an adjusted weighted sum of the neighbors ratings for that item [, 4]. In this paper, we introduce a new variance-based approach that works in conjunction with this particular neighborhood-based CF technique. The performance of recommender systems can be evaluated along several dimensions, such as the accuracy of recommendations for each user as well as the diversity of recommendations across different users [7]. We focus on the tradeoff between accuracy and diversity and explore several ways for overcoming it. With similar motivation, [] proposed product retrieval strategies (based on product queries) that are designed to improve diversity without compromising similarity. Various metrics can be used to measure the diversity of recommendations, such as the percentage of items that the recommender system is able to make recommendations for [7] or an average dissimilarity between all pairs of recommended items [3, ]. In this paper, we use the total number of distinct items recommended across all users as a diversity metric [2]. Several variance-based filtering approaches have been introduced to increase the accuracy of recommendations [2], but this improvement in accuracy comes at the expense of the diversity, confirming the inherent accuracy-diversity tradeoff. The approach presented in [2] works as a post-processing step (i.e., after the unknown ratings are predicted) for any existing recommendation technique based on all known ratings for each item. The rating variance of an item has also been used when recommender systems select items to be rated by new users, which would lead to more accurate subsequent predictions for these users [8, 9]. These methods find items with high variance and entropy, based on the assumption that an item that polarizes users interests provides more information about the user than an item that almost everyone liked. 3. Combining Rating Variance Information with Collaborative Filtering In contrast to [2], where the rating variance is calculated based on all known ratings of an item, in this paper we focus on neighborhood-based CF techniques, where the variance of each predicted rating can be calculated from the ratings of neighbors that were directly involved in the prediction of that rating. We first show some empirical evidence to support our motivation of using neighbors rating variance and then describe the algorithm of the proposed approach. 3. Motivation for Using Neighbors Rating Variance The premise that neighbors rating variance can affect the accuracy of recommendations is supported by the empirical results. In particular, we used a MovieLens ratings dataset (available at grouplens.org) with million movie ratings by various users. We pre-processed the data to include users and movies with significant rating history (i.e., users who rated at least 00 movies and movies rated by at least 00 users). We randomly chose 60% of the ratings for training, and calculated the predictions for the remaining 40% using the neighborhood-based CF technique. Figure (a) shows a clear pattern in a relationship between the rating variance of the neighbors used in the prediction of a particular rating and the accuracy of that rating measured by mean absolute error. The prediction error monotonically increases as the rating variance increases (i.e., when the closest 50

3 neighbors of the user have more divergent opinions about the predicted item). Thus, the variance of neighbors ratings can be an important factor in improving the recommendation accuracy. We also found that highly predicted items tend to be rated by more users, as shown in Figure (b). Because some of the highly predicted popular items tend to have relatively higher neighbor rating variance than less popular items (as we have observed in some of the data), recommending highly-predicted items with lower rating variance (i.e., less popular items) could improve the recommendation diversity. Number of Ratings 40K 20K 00K 80K 60K 40K 20K K Number of Ratings: user-based CF Number of Ratings: item-based CF Avg MAE: user-based CF Avg MAE: item-based CF Neighbors' Rating Variance.75-2 >= Avg MAE Avg Number of Ratings in Training Dataset User-based CF Item-based CF Predicted Rating Value (a) Average MAE and 50 Neighbors Rating Variance (b) Average Number of Ratings and Predicted Rating Value Figure. Some characteristic patterns of the CF recommendation technique. 3.2 Recommendation Ranking By Rating Variance Recommender systems typically recommend the most relevant N items to each user, since users are usually interested in only several highly-ranked item recommendations. The ratings of our datasets are integers between and 5, inclusive, where higher value represents a better-liked item. Accordingly, we can define the items with ratings 4 and 5 (i.e., greater then 3.5) as relevant and the ratings, 2, and 3 (i.e., less than 3.5) as non-relevant. We then evaluate the accuracy of recommendations based on the percentage of truly relevant ratings among those that were predicted to be the N most relevant items for each user, i.e., using the popular precision-in-top-n metric [7]. As mentioned above, recommendations are also evaluated using diversity metric, i.e., the total number of distinct items recommended across all users [2]. Algorithm. Integrating variance-based approach with neighborhood-based CF technique Input : Training data (60% of the dataset) and Test data (40% of the dataset) Output: Top N items (recommendations) for each user () Cosine-based similarity computation from Training data For each pair of two users u and u i I ( u, u ') R( u, i) R(, i) sim( u, ) = I ( u, ) : the set of all items rated by both users u and i I ( u, u ') R( u, i) 2 i I ( u, u ') R(, i) (2) Prediction For each pair of(u, i) in Test data K( u) sim( u, u ') R(, i) R( ) Rui (, ) = Ru ( ) + sim( u, u ') K( u) ( ) 2 ( ) K( u) R(, i) R( i) Var( u, i) = // variance of K neighbors ratings for item i K (3) Recommendation For each user u in Test data Filter the items by a predicted rating threshold (T:= 3.5, 3.6,.., 4.9) Rank the remaining items i by Var(u,i) Recommend top N (e.g.,, 3, 5) items to user u, according to their ranking 2 K ( u ) : K neighbors of user u who rated item i and have the highest similarity sim(u, ) to user u R (u ) : the average rating of user u R (i) : the average rating of all K neighbors on item i

4 The basic idea behind our new variance-based approach is that the items with the low neighbors rating variance should be recommended to a user, which could arguably lead to an improvement in both accuracy and diversity. Traditional recommender systems filter the predicted ratings that are greater than the pre-defined acceptable threshold (e.g., 3.5 out of 5), and then choose top N items with the highest predicted ratings. In contrast, based on our earlier empirical observations, the new approach chooses top N items according to the neighbors rating variance. However, the predicted rating value of an item is important and should be taken into account when filtering and ordering recommended items. The detailed algorithm is formally presented in Algorithm. A neighborhood-based CF technique can be user-based or item-based, depending on whether we calculate the similarity between users or items. We show only the user-based approach in Algorithm, but it can be straightforwardly rewritten for the item-based approach (because of the symmetry between users and items in all neighborhood-based CF calculations). In our experiments, both user- and item-based approaches were examined with varying neighborhood sizes. 4. Empirical Results The new approach with different predicted rating threshold T (e.g., T:= 3.5, 3.6,, 4.9) is evaluated in terms of precision- and diversity-in-top-n metrics and is compared with the standard CF approach which does not consider rating variance. As expected, the results in Figure 2 show an inverse relationship between accuracy and diversity. That is, the new approach with the lowest threshold (3.5) increases the diversity at the expense of accuracy and vice versa for the highest threshold (4.9). However, the configurations with the threshold between 3.9 and 4.3 (in the upper-right quadrant with respect to the standard CF approach, which is denoted by a triangle) represent significant improvements in both accuracy and diversity. We also note that, with higher predicted rating thresholds, the recommender system sometimes is not able to provide all N recommendations for each user. For example, about 70% of all possible item recommendations that can be obtained from a standard CF technique are used across all users by the proposed approach with the predicted rating threshold 4.3. It is interesting to see that, despite the smaller number of recommendations, the proposed approach still increases both the recommendation accuracy and diversity. This provides the recommender system designers with the flexibility to use other recommendation strategies for filling out the remaining top-n item slots and potentially improve the recommendation performance even further Accuracy Improvement Accuracy & Diversity Improvement Diversity Improvement Performance of Variance-based Approach with different Predicted Rating Threshold (T:=3.5~4.9) Performance of Standard CF Shaded Quadrant: represents the improvement in both Accuracy and Diversity In addition to the MovieLens dataset, we tested the proposed approach with other movie rating datasets, such as Netflix (dataset available at netflixprize.com) and Yahoo! Movies (ratings publicly displayed at movies.yahoo.com), and the same pattern is observed in the vast majority of cases (including both userand item-based CF technique in recommending top, 3, and 5 items for each user), as shown in Figure 3. Lastly, we also compared our new approach with an existing variance-based filtering approach (called the safe approach) that has been introduced to improve the accuracy of recommendations [2]. The results in Figure 4 show that the results of the safe approach are dominated by the proposed variance-based ranking technique. Furthermore, the results of the safe approach with different rating variance parameter values (k:= 0., 0.2,,.0) are all located in the upper left quadrant with respect to the standard CF approach, implying that a higher accuracy is obtained directly at the expense of the diversity. T=4.9 T=4.3 T=3.9 T=3.5 Figure 2. Precision- and diversity-in-top 5 items using the variance-based ranking approach. (MovieLens dataset, 50 neighbors, user-based CF)

5 MovieLens (50 Neighbors) Netflix (50 Neighbors) Yahoo! Movies (5 Neighbors) User-based CF technique Top Items Top 3 Items Top 5 Items Item-based CF technique Top Items Top 3 Items Top 5 Items Figure 3. Precision and diversity in top-n (N =, 3, 5) items using the variance-based ranking approach. 5. Conclusions In this paper, we proposed a new variance-based ranking approach that complements traditional neighborhood-based CF techniques and can overcome the accuracy-diversity tradeoff. Differently from the prior work, this approach takes advantage of this particular neighborhood characteristic by using the variance of neighbors ratings in recommendation processes. The new approach recommends top N items with low rating variance to each user as long as the items are regarded as relevant (i.e., the predicted ratings are above the pre-specified threshold). Using several movie rating datasets, the experimental results consistently show that the new approach within a certain range of predicted rating thresholds can generate the recommendations that are both more accurate and more diverse than the baseline CF recommendations. Recommender systems have made significant progress over the last few years; many techniques have been proposed to improve the performance of recommender systems. However, the ability for the

6 recommendation techniques to improve both accuracy and diversity at the same time remains largely unexplored. Also, we believe that rating variance can be used in the recommendation process in many different ways, and hope that this paper can stimulate more research on this and related topics MovieLens (50 Neighbors) User-based CF technique, Top 5 Items T=4.9 k= k= T=3.5 Performance of Variance-based Approach with different Predicted Rating Threshold (T:=3.5~4.9) Safe Approach with various portions of one rating standard deviation (k:=0.~) Performance of Standard CF 5 5 Netflix (50 Neighbors) Item-based CF technique, Top 0 Items T=4.9 k= k= T=3.5 Performance of Variance-based Approach with different Predicted Rating Threshold (T:=3.5~4.9) Safe Approach with various portions of one rating standard deviation (k:=0.~) Performance of Standard CF Figure 4. Comparison of the proposed variance-based ranking approach and the safe approach. Acknowledgments Work reported in this paper was supported in part by the National Science Foundation grant no References [] G. Adomavicius, A. Tuzhilin, Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions, IEEE Trans. on Knowledge and Data Engineering, 7(6): , [2] G. Adomavicius, S. Kamireddy, and Y. Kwon, Towards More Confident Recommendations: Improving Recommender Systems Using Filtering Approach Based on Rating Variance, Proc. of the 7 th Workshop on Information Technology and Systems, 2007 [3] K. Bradley and B. Smyth, Improving Recommendation Diversity, Proc. of 2 th Irish Conf. on Artificial Intelligence and Cognitive Science, 200. [4] J.S. Breese, D. Heckerman, C. Kadie, Empirical Analysis of Predictive Algorithms for Collaborative Filtering, Proceedings of 4 th Conference on Uncertainty in Artificial Intelligence, 998. [5] E. Brynjolfsson, Y J. Hu, and, D. Simester, Goodbye Pareto Principle, Hello Long Tail: The Effect of Search Costs on the Concentration of Product Sales, NET Institute Working Paper, [6] D. Fleder and K. Hosanagar, Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity, NET Institute Working Paper No. #07-0, [7] J.L. Herlocker, J.A. Konstan, L.G. Terveen, J. Riedl, Evaluating Collaborative Filtering Recommender Systems, ACM Transactions on Information Systems, 22():5-53, [8] A. Kohrs and B. Merialdo, Improving Collaborative Filtering for New Users by Smart Object Selection, Proc. of International Conference on Media Features (ICMF), 200. [9] A.M. Rashid, I. Albert, D. Cosley, S.K. Lam, S.M. McNee, J.A. Konstan, and J. Riedl, Getting to Know You: Learning New User Preferences in Recommender Systems, Proc. Int l Conf. Intelligent User Interfaces, [0] P. Resnick, H. R. Varian, Recommender systems, Comm. ACM, 40(3):56 58, 997. [] B. Smyth. and P. McClave, Similarity vs. Diversity, Proc. of the 4 th Intl. Conf. on Case-Based Reasoning: Case-Based Reasoning Research and Development, 200.