Lesson 6 Learning II Anders Lyhne Christensen, D6.05, INTRODUCTION TO AUTONOMOUS MOBILE ROBOTS

Transcription

1 Lesson 6 Learning II Anders Lyhne Christensen, D6.05, anders.christensen@iscte.pt INTRODUCTION TO AUTONOMOUS MOBILE ROBOTS

2 First: Quick Background in Neural Nets Some of earliest work in neural networks (or connectionist systems) was McCulloch-Pitts model of neurons (1943) McCulloch-Pitts: Simple linear threshold unit Synaptic weights associated with each synaptic input If threshold exceeded, neuron fired, carrying output to next neuron Later, Rosenblatt (1958) introduced Perceptron: Input vector x 1 x 2 x 3... x n w 1 w 2 w 3... w n Synaptic weights Neuron Σ θ Output

3 Neural Net Quick Background (con t.) 1960s 1970s: neural net research in decline due to Minsky and Papertbook (1969) that proved limitations of single-layer perceptron networks 1980s: resurgence due to multi-layer neural networks and use of back-propagation as a means for training these systems Much work in connectionism since, with significant progress Keep in mind: neural nets are only abstract computational models of biological neurons

4 Methods for Encoding Behavior-Based Robotic Control in Neural Networks HebbianLearning PerceptronLearning Classical Conditioning

5 1. Hebbian Learning Hebb(1949) developed one of earliest training algorithms for neural networks Hebbianlearning: increases synaptic strength along neural pathways associated with a stimulus and a correct response Specifically: w ij (t+1) =w ij (t) + η*o i o j Donald Hebb where: w ij (t) and w ij (t+1) are synaptic weights connecting neurons i and j before and after updating η is the learning rate coefficient o i ando j are the outputs of neurons iandj, respectively

6 Perceptron Learning has been used for Robotic Learning Overall training procedure: Repeat: 1. Present an example from a set of positive and negative learning experiences. 2. Verify the output of the network as to whether it is correct or incorrect. 3. If it is incorrect, supply the correct output at the output unit. 4. Adjust the synaptic weights of the perceptrons in a manner that reduces the error between the observed output and correct output. Until satisfactory performance as manifested by convergence is achieved or some other stopping condition is met.

7 How to Update Synaptic Weights? Delta rule: used for perceptrons without hidden layers Modify synaptic weights according to the formula: (w ij ) = η*w ij *(t j -o j ) where (w ij ) is the synaptic adjustment applied to the connection between neurons i and j η is the learning rate coefficient t j ando j are the correct and incorrect outputs, respectively TheDelta rule strives to minimize the error term using a gradient descent approach

8 2. Gradient Descent Approach Gradient Descent: Refers to learning methods that seek to minimize an objective function by which system performance is measured At each point in time, the policy is to choose the next step that yields the minimal objective function value. The learning rate parameters refer to the step size taken at each point in time Each step is computed only on the basis of local information, which is extremely efficient, but introduces possibility of traps in local minima Gradient Descent Hill-Climbing: Analogous process whereby objective function is maximized Hill climbing

9 Another Method for Updating Weights: Back-Propagation Back-propagation: most commonly used method for updating synaptic weights Employs generalized version of Delta rule for use in multilayer perceptron networks (which is commonly used in robotic control and vision) Usually, synaptic weights are initialized to random values Weights are adjusted by following update rule as training instances are provided: w ij (t+1) =w ij (t) + ηδ j o i where: δ j = o j (1-o j )(t j -o j ) for an output node, and δ j = o j (1-o j )Σ k δ k w jk for a hidden layer node t j ando j are the correct and incorrect outputs, respectively The errors are propagated backward from the output layer.

10 3. Classical Conditioning Studied by Pavlov (1927), assumes that unconditioned stimuli (US) automatically generates an unconditioned response (UR) Pavlov determined: US-UR pair is defined genetically and is appropriate to ensure survival in the agent s environment In Pavlov s studies: Sight of food (US) results in dog s salivation (UR) Associations can be developed between a conditioned stimulus (CS), which has no intrinsic survival value, and the UR Further studies: Bell rings repeatedly with sight of food Leads to bell ringing alone generating salivation Ivan Pavlov NOTE: Hebbian learning can also produce classical conditioning

11 Classical Conditioning for Self-Organization of Behavior-Based Robotic System Instead of hard-wiring relationships between stimuli and responses, learning architecture permits associations to develop over time SENSORS Aversive Unconditioned Stimulus Collision Detectors Range Finder Target Detector Conditioned Stimulus Inhibition Unconditioned Response Robot Motors Appetitive Unconditioned Stimulus

12 Genetic Algorithms (GA) GAs form a class of gradient descent methods in which a high-quality solution is found by applying a set of biologically inspired operators to individual points within a search space, yielding better generations of solutions over an evolutionary time scale Fitness of each member of the population (the set of points in the search space) is computed using an evaluation function, called fitness function, that measures how well each individual performs with respect to the task Population s best members are rewarded according to their fitness Poorly performing individuals are punished or deleted from the population Over generations, the population improves the quality of its set of solutions GAs are not guaranteed to yield an optimal global solution, they generally produce high-quality solutions within reasonable amounts of time for certain problem spaces (including learning of control strategies for behavior-based robots)

13 GAs (con t.) Genetic algorithms: usually require specialized knowledge representations (encodings) to facilitate their operator s operation Encodings: take form of position-dependent bit strings in which each bit represents a gene in the string chromosome Initial population established by some means, typically randomly Genetic operators applied to big string encoding of population members Three most frequently used operators: Reproduction Crossover Mutation

14 Genetic Operators Reproduction Crossover Mutuate

15 GAs for Robot Learning Prior to operator s application, each individual s fitness is computed using fitness function For robot learning, this may involve running a robot through a series of experiments, using the encoding of the behavioral controller represented by the particular individual bit string encoding being evaluated Fitness function returns a value capturing the robot s overall performance for the set of conditions being tested

16 GAs for Robot Learning (con t.) Reproduction operator: Fittest individuals are copied exactly and replace less-fit individuals This can bedone probabilistically, usually using roulette-wheel selection Net effect: increase in ratio of highly-fit individuals relative to # of poor performers Crossover operator: Exchange of information through transfer of info between two individuals Process creates new individuals that may or may not perform better than parents Which individuals to cross over and which bit string pairs to exchange is often done randomly Net effect: increase in overall population

17 GAs for Robot Learning (con t.) Mutation operator: Simple probabilistic flipping of bit values in encoding Affects individual only Probability of mutation is generally low Net effect: ability to escape local minima Overall effect: changing population over time Final quality and length of time to obtain solution depend on nature of problem and parameters used

18 Example of GAs for Learning Behavioral Control GAs, although powerful, require some restrictions on implementation compared to previous learning approaches we ve discussed Much of learning must be done off-line, since: Significant population of robots needed for fitness testing Robots must be tested over many, many generations Simulations, fortunately, can be run much faster than real-world testing If simulation has reasonable fidelity to real robot and environment, control parameters from fittest simulated individual can be transferred to actual robot for use

19 Example Robot GA Code begin Obstacles.Create; Population.Build; for 1 to NUMBER_GENERATIONS do begin for 1 to RUNS_PER_GENERATION do begin foreach ROBOT in Population do for 1 to MAX_NUMBER_STEPS do begin ROBOT.Move end Obstacles.Recreate; end end Robots.Reproduce; Robots.Crossover; Robots.Mutate; end end

20 Hybrid Genetic/Neural Learning and Control Several researchers have combined neural nets and GAs for robot learning Typically, approach is to use GAs to learn synaptic weights for a neural controller Example by Floreano et al.: Implementation of Braitenberg-style neural controller Use of robot (Khepera) with 3 ambient light sensors and 8 IR proximity sensors Fitness functions defined for behaviors, including: Navigation and obstacle avoidance: Fitness maximizes motion and distance from obstacles Homing: Fitness ensures that power is kept at adequate levels by adding a lightseeking behavior to guide it to its black recharging area when power becomes low Grasping of balls using an added gripper: Fitness maximizes the number of objects gripped in an obstacle-free environment Most successful individual: learned to back up until it encountered something, then turned around an attempted to grip it

21 Today s task Work on projects