Exploiting Similarity to Optimize Recommendations from User Feedback

Transcription

1 1

2 Exploiting Similarity to Optimize Recommendations from User Feedback Hasta Vanchinathan Andreas Krause (Learning and Adaptive Systems Group, D-INF,ETHZ ) Collaborators: Isidor Nikolic (Microsoft, Zurich), Fabio De Bona (Google, Zurich) 2

3 A Recommendation Example 3

4 A Recommendation Example 4

5 A Recommendation Example 5

6 A Recommendation Example 6

7 A Recommendation Example 7

8 A Recommendation Example 8

9 A Recommendation Example 9

10 A Recommendation Example 10

11 Many real world instances Disclaimer: All trademarks belong to respective owners

12 Many real world instances Disclaimer: All trademarks belong to respective owners

13 Many real world instances Disclaimer: All trademarks belong to respective owners

14 Many real world instances Disclaimer: All trademarks belong to respective owners

15 Many real world instances Disclaimer: All trademarks belong to respective owners

16 Many real world instances Disclaimer: All trademarks belong to respective owners

17 Many real world instances Disclaimer: All trademarks belong to respective owners

18 Many real world instances Disclaimer: All trademarks belong to respective owners

19 Common Thread 19

20 Common Thread To do well, we need a model. e.g., 20

21 Common Thread To do well, we need a model. e.g., Popular techniques include Content-based filtering Collaborative filtering Hybrid recommendation systems 21

22 Common Thread To do well, we need a model. e.g., Popular techniques include Content-based filtering Collaborative filtering Hybrid recommendation systems All aim to predict reward given a fixed data set 22

23 Challenges 23

24 Challenges Many, dynamic! 24

25 Challenges Many, dynamic! Preferences change 25

26 Challenges Many, dynamic! Estimating all combinations both hard and wasteful! Preferences change 26

27 Challenges Many, dynamic! Estimating all combinations both hard and wasteful! Preferences change Only need identify high reward items! 27

28 Challenges Many, dynamic! Estimating all combinations both hard and wasteful! Preferences change Only need identify high reward items! 28

29 Multi Arm Bandits 29

30 Multi Arm Bandits 30

31 Multi Arm Bandits Early approaches require k << T 31

32 Multi Arm Bandits Early approaches require k << T Can get strong guarantees for a finite set of actions Gittins indices - Greedy, UCB1 (Auer et al 01) #of arms increases -> performance degrades 32

33 Multi Arm Bandits Early approaches require k << T Can get strong guarantees for a finite set of actions Gittins indices - Greedy, UCB1 (Auer et al 01) #of arms increases -> performance degrades For dynamic web scale recommendations, k >> T 33

34 Learning meets bandits f(x) x 34

35 Learning meets bandits Exploit similarity information to predict rewards for new items f(x) x 35

36 Learning meets bandits Exploit similarity information to predict rewards for new items Must make assumptions on reward function, e.g.: f(x) x 36

37 Learning meets bandits Exploit similarity information to predict rewards for new items Must make assumptions on reward function, e.g.: Linear (linucb - Li et al 10) f(x) x 37

38 Learning meets bandits Exploit similarity information to predict rewards for new items Must make assumptions on reward function, e.g.: Linear (linucb - Li et al 10) Lipschitz (Bubeck et al 08) f(x) x 38

39 Learning meets bandits Exploit similarity information to predict rewards for new items Must make assumptions on reward function, e.g.: Linear (linucb - Li et al 10) Lipschitz (Bubeck et al 08) Low RKHS norm (GP-UCB - Srinivas et al 12) f(x) x 39

40 Learning meets bandits Exploit similarity information to predict rewards for new items Must make assumptions on reward function, e.g.: Linear (linucb - Li et al 10) Lipschitz (Bubeck et al 08) Low RKHS norm (GP-UCB - Srinivas et al 12) This is the approach we pursue in this work! f(x) x 40

41 Problem Setup 41

42 Problem Setup 42

43 Problem Setup = user attributes 43

44 Problem Setup = user attributes 44

45 Problem Setup = user attributes 45

46 Problem Setup = user attributes 46

47 Problem Setup = user attributes 47

48 Problem Setup = user attributes 48

49 Problem Setup = user attributes 49

50 Problem Setup = user attributes 50

51 Problem Setup = user attributes We want to maximize: 51

52 Problem Setup = user attributes Equivalently, minimize 52

53 Problem Setup = user attributes Equivalently, minimize 53

54 Our Approach 54

55 Our Approach We propose CGPRank, that uses a bayesian model for the rewards 55

56 Our Approach We propose CGPRank, that uses a bayesian model for the rewards CGPRank efficiently shares reward across 56

57 Our Approach We propose CGPRank, that uses a bayesian model for the rewards CGPRank efficiently shares reward across Items 57

58 Our Approach We propose CGPRank, that uses a bayesian model for the rewards CGPRank efficiently shares reward across Items Users 58

59 Our Approach We propose CGPRank, that uses a bayesian model for the rewards CGPRank efficiently shares reward across Items Users positions 59

60 Demux ing Feedback 60

61 Demux ing Feedback We still need to predict: 61

62 Demux ing Feedback We still need to predict: Assume: items do not influence reward of other items 62

63 Demux ing Feedback We still need to predict: Assume: items do not influence reward of other items 63

64 Demux ing Feedback We still need to predict: Assume: items do not influence reward of other items 64

65 Demux ing Feedback We still need to predict: Assume: items do not influence reward of other items relevance! 65

66 Demux ing Feedback We still need to predict: Assume: items do not influence reward of other items relevance! Position CTR! 66

67 CGPRank Sharing across positions 67

68 CGPRank Sharing across positions 68

69 CGPRank Sharing across positions

70 CGPRank Sharing across positions

71 CGPRank Sharing across positions ?? 0.16??

72 CGPRank Sharing across positions ?? 0.16??

73 CGPRank Sharing across positions ?? Position weights - independent of items! - estimated from logs 0.16??

74 CGPRank Sharing across positions Position weights - independent of items! - estimated from logs

75 CGPRank Sharing across items/users 75

76 CGPRank Sharing across items/users 76

77 CGPRank Sharing across items/users 77

78 CGPRank Sharing across items/users 78

79 CGPRank Sharing across items/users 79

80 CGPRank Sharing across items/users 80

81 CGPRank Sharing across items/users 81

82 CGPRank Sharing across items/users 82

83 CGPRank Sharing across items/users 83

84 CGPRank Sharing across items/users 84

85 CGPRank Sharing across items/users 85

86 CGPRank Sharing across items/users 86

87 CGPRank Sharing across items/users 87

88 Sharing across items / users with Gaussian processes Bayesian models for functions Prior P(f) f(x) reward x choice 88

89 Sharing across items / users with Gaussian processes Bayesian models for functions Prior P(f) f(x) reward x choice 89

90 Sharing across items / users with Gaussian processes Bayesian models for functions Prior P(f) f(x) reward x choice 90

91 Sharing across items / users with Gaussian processes Bayesian models for functions Prior P(f) f(x) reward likely x choice 91

92 Sharing across items / users with Bayesian models for functions Gaussian processes Prior P(f) unlikely f(x) reward likely x choice 92

93 Sharing across items / users with Gaussian processes Bayesian models for functions Likelihood P(data f) Prior P(f) unlikely f(x) f(x) reward likely x choice 93

94 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely x choice x 94

95 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely x choice x 95

96 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely likely x choice x 96

97 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely likely x choice x 97

98 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely likely x choice x unlikely 98

99 Sharing across items / users with Bayesian models for functions Prior P(f) Gaussian processes unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely likely x choice x unlikely Closed form Bayesian posterior inference possible! 99

100 Sharing across items / users with Gaussian processes Bayesian models for functions Prior P(f) unlikely f(x) Likelihood P(data f) Posterior: P(f data) f(x) reward likely likely x choice Closed form Bayesian posterior inference possible! Allows to represent uncertainty in prediction x unlikely 100

101 Predictive confidence in GPs f(x) Typically, only care about marginals, i.e., x 101

102 Predictive confidence in GPs f(x) Typically, only care about marginals, i.e., x x 102

103 Predictive confidence in GPs f(x) f(x ) x x Typically, only care about marginals, i.e., P(f(x )) 103

104 Predictive confidence in GPs f(x) f(x ) x x Typically, only care about marginals, i.e., P(f(x )) Parameterized by covariance function K(x,x ) = Cov(f(x),f(x )) 104

105 Predictive confidence in GPs f(x) f(x ) x x Typically, only care about marginals, i.e., P(f(x )) Parameterized by covariance function K(x,x ) = Cov(f(x),f(x )) Can capture many rec. tasks using appropriate cov. function 105

106 Intuition: Explore-Exploit using GPs Selection Rule: 118

107 Intuition: Explore-Exploit using GPs Selection Rule: 119

108 CGPRank Selection Rule 120

109 CGPRank Selection Rule At t=0, if no prior observations 121

110 CGPRank Selection Rule At t=0, with some prior observation 122

111 CGPRank Selection Rule Uncertainty shrinks not just at observation. 123

112 CGPRank Selection Rule but also at other locations based on similarity! 124

113 CGPRank Selection Rule If list size is 2 125

114 CGPRank Selection Rule The first item,, is selected according to 126

115 CGPRank Selection Rule 127

116 CGPRank Selection Rule Secret sauce? 128

117 CGPRank Selection Rule Time varying tradeoff parameter 129

118 CGPRank Selection Rule Hallucinate mean and shrink uncertainties 130

119 CGPRank Selection Rule Hallucinate mean and shrink uncertainties 131

120 CGPRank Selection Rule Now update model and again pick using: 132

121 CGPRank Selection Rule Now update model and again pick using: 133

122 CGPRank 134

123 CGPRank 135

124 CGPRank 136

125 CGPRank 137

126 CGPRank 138

127 CGPRank 139

128 CGPRank 140

129 CGPRank 141

130 CGPRank 142

131 CGPRank 143

132 CGPRank 144

133 CGPRank 145

134 CGPRank 146

135 Theorem 1 If we choose CGPRank - guarantees, then running CGPRank for T rounds, we incur a regret sublinear in T. Specifically, Grows strongly sublinearly for typical kernels 147

136 Experiments - Datasets 153

137 Experiments - Datasets Google book store logs 42 days of user logs Given key book, suggest list of related books Kernel computed from related graph on books 154

138 Experiments - Datasets Google book store logs 42 days of user logs Given key book, suggest list of related books Kernel computed from related graph on books Yahoo! Webscope R6B* 10 days of user log on Yahoo! Frontpage Unbiased method to test bandit algorithms 45 million user interations with 271 articles Feedback available for single selection, we simulated list selection 155

139 Experiments - Questions How much does principled sharing of feedback help? Across items/context? Across positions? Can CGPRank outperform an existing, tuned recommendation system? 156

140 Sharing across items 157

141 Sharing across contexts 158

142 Effect of increasing list size 159

143 Boost over existing approach Existing Algorithm 160

144 Conclusions CGPRank - Efficient Algorithm with strong theoretical guarantees Can generalize from sparse feedback across Items Contexts Positions Experiments suggest Statistical and computational efficiency 161