Training deep Autoencoders for collaborative filtering Oleksii Kuchaiev & Boris Ginsburg

Transcription

1 Training deep Autoencoders for collaborative filtering Oleksii Kuchaiev & Boris Ginsburg

2 Motivation Personalized recommendations 2

3 Key points (spoiler alert) 1. Deep autoencoder for collaborative filtering 1. Improves generalization 2. Right activation function (SELU, ELU, LeakyRELU) enables deep architectures 1. No layer-wise pre-training, or skip connections 3. Heavy use of dropout 4. Dense re-feeding for faster and better training 5. Beats other models on time-split Netflix data (RMSE of vs ) 6. (PyTorch-based) Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 3

4 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 4

5 Collaborative filtering Rating prediction R(i,j) = k iff user i gave item j rating k j Some of the most popular approaches Alternative Least Squares (ALS) 3 X Items r 4 m users R=Rating matrix Users i n items r hidden factors 5

6 Autoencoder Deep learning tool of choice for dimensionality reduction E n c o d e r y x z = e 2 e 2 = f(w e 2 e 1 + b 2 ) e 1 = f(w e 1 x + b 1 ) d 2 = W d 1 d 1 + b 4 d 1 = f(w d 2 z + b 3 ) D e c o d e r z = encoder(x), encoding y = decoder(z), reconstruction of x y = decoder(encoder(x)) Autoencoder can be thought of as generalization of PCA Constrained if decoder weights are transpose of encoder De-noising if noise is a added to x. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786),

7 AutoEncoders for recommendations User (item) based Masked Mean Squared Error z (very) sparse r dense y Sedhain, Suvash, et al. "Autorec: Autoencoders meet collaborative filtering." Proceedings of the 24th International Conference on World Wide Web. ACM,

8 Dataset Netflix prize training data set Time split to predict future ratings Wu, Chao-Yuan, et al. "Recurrent recommender networks." Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM,

9 Benchmark Netflix prize training data set RMSE = σ r i 0 r i y i 2 σ ri 0 1 PMF: Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems I-AR, U-AR: Sedhain, Suvash, et al. "Autorec: Autoencoders meet collaborative filtering." Proceedings of the 24th International Conference on World Wide Web. ACM, RRN: Wu, Chao-Yuan, et al. "Recurrent recommender networks." Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM,

10 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 10

11 RMSE Activation function matters - We found that on this task ELU, SELU and LRELU perform much better than SIGMOID, RELU, RELU6, TANH and SWISH Apparently important: a) non-zero negative part b) Unbounded positive part Training RMSE per mini-batch. All lines correspond to 4-layers autoencoder (2 layer encoder and 2 layer decoder) with hidden unit dimensions of 128. Different line colors correspond to different activation functions. Iteration 11

12 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 12

13 Overfit your data Wide layers generalize poorly RMSE RMSE d size 128 d size 256 d size 512 d size 1024 y d 2 = W 1 d d 1 + b 4 d e 1 = f(w 1 e x + b 1 ) x Evaluation RMSE > 1.1 on Netflix full d size 128 d size 256 d size 512 d size Epoch 13

14 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 14

15 Deeper models Generalize better y No layer-wise pre-training necessary! x 15

16 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 16

17 RMSE y Dropout Helps wider models generalize dropout Evaluation RMSE Drop Prob 0.0 Drop Prob 0.5 Drop Prob 0.65 Drop Prob x Epoch 17

18 Autoencoders & collaborative filtering Effects of the activation types Overfitting the data Going deeper Dropout Dense re-feeding Conclusions Oleksii Kuchaiev and Boris Ginsburg "Training Deep AutoEncoders for Collaborative Filtering, arxiv preprint arxiv: (2017). 18

19 Dense re-feeding Note that x is sparse but f(x) is dense For x, most of the loss is masked Intuition: idealized scenario Imagine perfect f x i 0: f x i = x i If user later rates new item k with rating r, then: f x k = r f(x)= By induction: f f x = f x Thus, f(x) should be a fixed point of f for every valid x 19

20 Dense re-feeding Attempt to enforce fixed point constraint (very) sparse x Dense f(x) Dense f(x) Dense f(f(x)) Update with real data x Update with synthetic data f(x) 20

21 RMSE Dense re-feeding Together with bigger LR improves generalization Baseline Baseline LR Baseline RF Baseline LF RF Epoch 21

22 Results Netflix time split data I-AR, U-AR: Sedhain, Suvash, et al. "Autorec: Autoencoders meet collaborative filtering." Proceedings of the 24th International Conference on World Wide Web. ACM, RRN: Wu, Chao-Yuan, et al. "Recurrent recommender networks." Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, DeepRec is our 6 layer model 22

23 Conclusions 1. Autoencoders can replace ALS and be competitive with other methods 2. Deeper models generalize better 1. No layer-wise pre-training is necessary 3. Right activation function enables deep architectures 1. Negative parts are important 2. Unbounded positive part 4. Heavy use of dropout is needed for wider models 5. Dense re-feeding further improves generalization 23

24 Oleksii Kuchaiev and Boris Ginsburg "Training Deep Autoencoders for Collaborative Filtering, arxiv preprint arxiv: (2017). Code, docs and tutorial: