March 12, Introduction to reinforcement learning. Pantelis P. Analytis. Introduction. classical and operant conditioning.

Transcription

1 March 12, / 27

2 / 27

3 What s? 3 / 27

4 What s? 4 / 27

5 classical Conditioned stimulus (e.g. a sound), unconditioned stimulus (e.g. the taste of food), unconditioned response (unlearned behavior such as salivation). 5 / 27

6 Behaviorism in psychology Psychology was under the grip of behaviorism from the 20s the 60s. Focus on expressed behavior rather than on psychological processes. 6 / 27

7 The Rescola-Wanger model V n+1 X = α X β(λ V t ) V n+1 X = V n X + V n+1 X V X is the change in the strength, on a single trial, of the association between the CS labelled X and the US α is the salience of X (bounded by 0 and 1) β is the rate parameter for the US (bounded by 0 and 1), sometimes called its association value λ is the maximum possible for the US V X is the current associative strength of X V t is the tal associative strength of all stimuli present, that is, X plus any others 7 / 27

8 The Rescola-Wanger model: predictions The model captures acquisition and extinction of associations through a process of surprise. First model incorporate several cues. Importantly, the model captures interactions between cues. One cue may block the association of another with the US. Extinction might not occur if an inhibir is there. Over time the model converges optimal least square weights. Examples: Blocking, overshadowing and weakening of stimuli. 8 / 27

9 The first experiments Thorndike studied the time that animals ok escape from his illustrious box. 9 / 27

10 Thorndike s law of effect Thorndike s law of effect: Of several responses made the same situation, those which are accompanied or closely followed by satisfaction the animal will, other things being equal, be more firmly connected with the situation, so that, when it recurs, they will be more likely recur; those which are accompanied or closely followed by discomfort the animal will, other things being equal, have their connections with that situation weakened, so that, when it recurs, they will be less likely occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond (Thorndike, 1911, p.244). 10 / 27

11 Learned helplessness The organisms learn that it is impossible escape, and even when the hindrance is removed they do not attempt escape. 11 / 27

12 The first experiments Operant can be described as a process that attempts modify behavior through the use of positive and negative. Through, an individual makes an association between a particular behavior and a consequence (Skinner, 1938). 12 / 27

13 Tolman s cognitive maps 3 groups of rats, running in a maze for 17 days. one group got a reward, the second got no reward, the third got a reward on the 11th day. 13 / 27

14 Implicit The group that was rewarded only on the 11th day improved rapidly and surpassed in terms of performance the group that was rewarded from the beginning. 14 / 27

15 There are two strategies solve RL problems. Organism can memorize rewards or construct a contingency map and plan ahead behavior. 15 / 27

16 The Iowa gambling task (Bachara et al. 1997) Participants are presented 4 decks on the computer and they are ld that each deck will reward them or penalize them. 100 trials in tal, unbeknownst the participants. The participants started with $ 2000 and are asked maximize their returns. 16 / 27

17 The Iowa gambling task Participants are presented 4 decks on the computer and they are ld that each deck will reward them or penalize them. Deck s A and B bring higher bring higher immediate rewards, but have negative expected value, while C and D have lower immediate rewards but positive expected value. 17 / 27

18 : expectation The delta rule is a popular model-free rule: E j (t) = E j (t 1) + δ j (t)η[r j (t) E j (t 1)], where δ j (t) is an indicar variable, being 1 if alternative j was chosen on trial t, and 0 otherwise. We opted for a simple fixed rate, η / 27

19 : expectation The decay rule is another popular model-free rule, according which expected values of the unchosen alternatives decay wards 0 (e.g. Erev and Roth, 1998): E j (t) = ηe j (t 1) + δ j (t)r j (t), with decay parameter 0 η / 27

20 : choice rules ɛ-greedy rule P(C(t) = j) = { (1 ɛ)/k max if E j (t) > E k (t), k j ɛ/(k K max ) otherwise where K is the number of arms and K max is the number of arms with the same maximum value. Softmax P(C(t) = j) = exp(θe j (t)) K k=1 exp(θe k(t)) 20 / 27

21 The Iowa gambling task: behavioral results Participants are presented 4 decks on the computer and they are ld that each deck will reward them or penalize them. Deck s A and B bring higher bring higher immediate rewards, but have negative expected value, while C and D have lower immediate rewards but positive expected value. 21 / 27

22 The Iowa gambling task: simulating models The models were fitted on data using maximum likelihood estimation. 22 / 27

23 Prediction competitions 23 / 27

24 Replicating well known findings 24 / 27

25 Studying widely used websites Can you develop a model of likes and comments on Instagram or Twitter? How does attention interact with liking in websites like Facebook? 25 / 27

26 Using big data from KDD competitions KDD regularly organizes competitions. Data from past events are available online. 26 / 27

27 Dataset reposiries 27 / 27