Latent Space Based Text Generation Using Attention Models

Transcription

1 Latent Space Based Text Generation Using Attention Models Jules Gagnon-Marchand Prepared for NLP Workshop for MILA Aug. 31, 2018

2 Introduction: Motivation Text Generation is important: Any AI based task with a linguistic output: Translation Question Answering Conversational Agents Text summarization Smart chat/ answer suggestion More that we will soon invent In all of these tasks, you want nice, human sounding text. Page 2

3 Introduction: Attention Models Attention-only models are recent v Introduced by Decomposable Attention Models for Natural Language Inference by Parikh et al., 2016 v Made really famous by Transformer, by Vaswani et al., àcomplete perceptual field for each layer à Trained much more quickly: Compared to RNN-models, with input feeding, no sequential structure at training time, making it much easier to parallelize à Transformer for example, takes 2 or 3 days on 8 GPUs, versus weeks for GNMT Page 3

4 Advantages of Attention Models Recently, at the top of a large number of very tasks in NLP : Neural Machine Translation: Google s Transformer & recently, Google s Universal Transformer, Harvard s Variational Transformer Question Answering: Allen Institute s Bidirectional Attentive Flow and Stanford s QANets Large number of Natural Language Understanding tasks: Open AI s fine tuned Transformer LM. Language Modelling: Decoder-only transformer, 64 layers, character based (Google AI) Success in Non NLP fields: Self Attention GAN: Image generation, from Ian Goodfellow s group Non-Local Networks: Strong performance in Video tasks. Graph Attention Networks (GAT): Deep learning to graph node classification Page 4

5

6 Introduction: Modern Attention Models à Translation: Transformer (Vaswani et al., 2017, Google Brain) Better performance Much smaller training costs Page 6

7 Background: Transformer Multi-Head Attention n Multi-head attention: Page 7

8 Background: Transformer Positional Encodings n Positional Encodings Self-attention doesn t know the positions, not even relative (CNNs know relative positions, RNNs know both) Related to CoordConv by Uber this year for CNNs They add an encoding to the word embeddings: with pos the position, i the dimension and d the number of dimensions They also experimented with learned encodings (positional embeddings), but the performance was similar. Chose to go with non-learned ones, as it generalizes to lengths not seen (or unfrequently seen) in training. Chose this function because in theory the model can learn to attend to relative positions, as PE pos + k can be represented as a linear function of PE pos + k Page 8

9 Introduction: Modern Attention Models à Graphs Networks: Graph Attention Networks (Velickovic et al., 2018, Cambridge + Univ. Autònoma de Barcelona + MILA) à Pure masked self-attention à Each node can do self-attention over it s neighbors. The other nodes are masked. Page 9

10 Introduction: Modern Attention Models à Conditional Image Generation : SAGAN (Zhang et al., 2018, Goodfellow s Group - Google Brain) à ResNet + Self-Attention Layer Page 10

11 Introduction: Modern Attention Models State of the Art for many NLU tasks: Improving Language Understanding with Unsupervised Learning (Radford et al., 2018, OpenAI) à 30 Layers deep Transformer model à All decoder layers à 1 month on 8 GPUs à Training in unsupervised fashion in Language model configuration: Predict the next word à Fine-tuned More details on the tasks: Page 11

12 Introduction: Modern Attention Models OpenAI s Transformer Fine-Tuning Process Page 12

13 Introduction: Modern Attention Models Page 13

14 Introduction: Modern Attention Models à Question Answering: QANet (Yu et al., 2018, CMU + Google Brain) Page 14

15 Background: Explaining the Attention Layer Page 15

16 Background: Neural Text Generation Different Families of Approaches for Neural Text Generation: We are here. (VAEs are also latentspace based) Page 16

17 Background: Neural Text Generation Page 17

18 Latent Space Based Methods ARAE: AAE: VAE: Page 18

19 Background : Adversarially Regularized Autoencoders (ARAE) Original ARAE: (Encoder + Generator) vs Discriminator Zhao et al., 2018 Modified ARAE: Generator vs (Discriminator + Encoder) Spinks et al., 2018 Page 19

20 Background: ARAE Decoder input noise Problem: The data (latent spaces) to generate is still close to discrete. Solution (still experimenting with): Decoder input noise. Page 20

21 Background: Adversarial Autoencoders Makhzani et al., 2015 Page 21

22 Background: Variational Autoencoders Evidence Lower Bound (ELBO): Page 22

23 n Proposed Solutions: Proposed Solutions: Improve on existing pure GAN and VAE methods: Attention models: Transformer Auto encoder Attention based generator and discriminator» Positional Encodings» Optional Multi-head attention Our Improved ARAE: State of the art in GAN stability: Spectral Normalization (widely accepted) Hinge Loss Our Improved AAE: Our Improved VAE: Page 23

24 Attention Generator and Discriminator Page 24

25 Results Numerical Values Quantitative Metrics using the Stanford Natural Language Inference Dataset: Reverse Perplexity Perplexity Real Data LM Samples Original ARAE Transformer ARAE Transformer AAE Lower is better. Page 25

26 Solutions n ARAE: Full codes Transformer Auto-Encoder Self-Attention based generators and discriminators Spectral Normalization (GAN) Positional Encodings Optional Multi-head attention Fixed Size Code Transformer Transformer Block Generator And Discriminator Page 26

27 Our Improved ARAE a man from the bus stop line up to board his delivery from the bus. a woman is looking at herself in her kitchen looking at framed photos. a man is watching the water practice. a man has his face met before sticking his tongue out. a woman is working on her baby. the woman is a the musician in the wedding. a man is carrying the counter of various items. the man in the shopping center is also in the car. the men are running from the parade. a lady is outside with a cat in a red jacket. the dogs are running through the water and the last lady in the comfort of the birds. the man is wearing a white shirt the lady and child are building some store items. a woman is speaking in front of a crowd at an overhead wall. two children are running for free falling leaves. a man on the train has a big green horse. a man is cleaning a boat of firefighters. a choir in a musical is in a half crowded auditorium. the couple is a performer on the cheek. a young person holds the flag, a dress from the bar. the man is sweeping the bathroom. the baby girl is at the baby-themed pool Experimental details: - Dataset: Google Sentence Compression (as in Cifka et al.) - Tokenizer: Google SentencePiece, BPE mode, vocabulary size of 16k subtokens (as in Cifka et al.) - Sentence Length: 20 BPE subtokens - Code depth: 304 (Cifka et al. use 300. Transformer s multi-head attention requires multiples of the number of heads, is the closest multiple of 8 to 300.) - Z dim: 100 (as in Cifka et al.) - Loss: ARAE, gradient l2 normalized to 1 as in Cifka et al., and then multiplied by the lambda - ARAE Lambda: Spectral Normalization: On GAN (not AE) - Encoder Output: - Full codes (no pooling) - Spherical normalization - Decoder Input: Addition of Gaussian noise of stddev Transformer - Number of attention heads: 8 - Transformer - Number of layers: (Blocks are as per the original paper) - 3 Encoder Blocks (Self-Attention -> Element-wise feedforward) - 3 Decoder Blocks (Masked Self-Attention -> Encoder-Decoder Attention -> Element-wise feedforward) Page 27

28 Solutions - Models n Our improved VAE: Transformer Auto-Encoder Average pooling or attention bottleneck Global average pooled Transformer Page 28

29 Comparison and Experiments: VAE Our improved VAE: The two young man and child are getting ready to eat. Two men walking on a red shore. The dog is walking with a bright train. A guy holds his head in the yard. A few people are getting ready off of her house. The two men are having a haircut. A girl is standing in the crowd with her bed. An old woman sits on a snowy hill. A man is taking a race during the park A young girl is looking at his stuff in a church. A woman is getting ready for her shoes. There are a young boy in the beautiful car. The people are performing in the snow. A child is running through the beach. Three men play underneath a big ball. The two men are walking through an office. The son were going to catch a video games. Experimental details: - Dataset: SNLI (as in Cifka et al.) - Tokenizer: Google SentencePiece, BPE mode, vocabulary size of 16k subtokens (as in Cifka et al.) - Sentence Length: 30 BPE subtokens - Code depth: 304 (Cifka et al. use 300. Transformer s multi-head attention requires multiples of the number of heads, is the closest multiple of 8 to 300.) - Loss: Original VAE loss - VAE Lambda: Encoder Output: - Attention Pooling to single 304 wide vector. - Transformer - Number of attention heads: 8 - Transformer - Number of layers: - 3 Encoder Blocks (Self-Attention -> Element-wise feedforward) - 3 Decoder Blocks (Masked Self- Attention -> Encoder-Decoder Attention -> Element-wise feedforward) Page 29

30 Solutions: Models n AAE: Transformer Auto-Encoder Self-Attention based discriminator (full codes) Average pooling or attention bottleneck otherwise Spectral Normalization Full Fix Sized Code Transformer Transformer Block Discriminator Page 30

31 Comparison and Experiments: AAE Our Improved AAE: a man is hanging from a cart. the two boys are not enjoying the scenery of the nearby mountains. a person has a bag around their head. a big party with two men and one woman. a woman uses a man as a teenage girl. a man in a white shirt playing a trumpet. a man is strumming on his guitar. two men and a woman are at a playground. the man is walking away from the camera. a young child is playing the piano. a man is browsing a business with a newspaper. the family is playing on the floor. a boy in a blue shirt and blue jeans is walking down a path. two men sitting on the floor in the summer. a young girl is looking at a horse. a boy wearing a shirt is raking leaves. the group of people are holding their hands across the field. the woman is wearing a black shirt while playing with a child. two men are outside and the girl is in her home. a man is training his dog to surf. a group of children riding scooters together down the road. a man in a chef's shirt is on the side of a counter. the man is cleaning the gutters from the older woman. a man in a hat is reading something on a bench. a woman in a black hoodie cleaning a window. woman drinking a beer men in orange vests and black pants push a yellow bus. the quarterback is preparing to throw a ball during a game. two people are standing in a crowd as they look out a window. a boy is swimming in the ocean. Experimental details: Dataset: SNLI Decoder input noise: 0.5 Bottleneck: Full encoder codes Discriminator: Self-Attention Num. Layers Discriminator: - 2 simplified transformer blocks - Global avg. pooling layer - Fully connected Sentence length: 30 BPE Code depth: 304 Num. of transformer layers: - 3 blocks for encoder - 3 blocks for decoder Page 31

32 Thank You! Jules Gagnon-Marchand Prepared for NLP Workshop for MILA Aug. 31, 2018