Motivational Behavior of Neurons and Fuzzy Logic of Brain (Can robot have drives and love work?)

Transcription

1 Motivational Behavior of Neurons and Fuzzy Logic of Brain (Can robot have drives and love work?) UZIEL SANDLER Jerusalem College of Technology Department of Applied Mathematics Jerusalem ISRAEL LEV TSITOLOVSKY Bar-Ilan University Department of Life Science Ramat Gan ISRAEL Abstract: Many theories relate to the brain as a complex network of neurons, which are approximated as simple elements that make summation of excitations and generate output reaction in accordance with simple activation functions. Such an idealization, however, is far from the properties of a real neuron. In this paper we present results of the experiments of a real neuron s learning and theoretical description this processes is based on Fuzzy Dynamics: contemporary theory operates with perceptions as with a mathematical object. We point out that logic of a neuron s decision-making may be close to the fuzzy logic. In the conclusion, we discuss possibility of design of a feeling robot, which will be able to use a trial-and-error self-learning process based on artificial motivations and effectively adapt (like an animal) its behavior in suddenly changing environmental conditions. 1 Introduction A brain does not possess complete information about the environment and its decision-making depends not only on information about a given event but also on the brain s initial state, in other words, on particular drive or motivation, i.e. forces acting either on or within a person to initiate behavior (we use words drive and motivation as synonyms, because on the neuronal level they are equivalent). There is a limited number of simple biologic drives: feeding, drinking, respiration, temperature regulation, sexual motivation, avoidance of danger and drug-dependence. Note, that these drives are not independent and can influence each other. For a detailed description of the general properties of motivations, we refer the reader to reviews cited in [1]. Motivation is satisfied by reward or by the avoidance of punishment (negative reinforcement). It is important that stability of internal environmental factors and homeostasis are associated with motivation. A deviation from the optimum induces action generation directed toward the correction of this deviation. For example, respiratory motivation is connected with a decrease in ph, thirst is an accompaniment of cell dehydration, strong adverse stimuli have an excitotoxic effect and the metabolic signal for feeding motivation is an energy depletion. So one can say that motivations arise as a result of deviations of vitally important constants from their optimal values and are related to transient damage or to the threat of damage to the organism. Note that strong excitation of the neurons in many cases lead to damage and death, so preservation from excitation usually protects neurons from damage. Moreover, simple biological motivations are connected with excitotoxicity and neuronal injury. The substances that decrease motivation, such as cholecystokinin, opiates, insulin, gamma-aminobutyric acid and dopamine, in the most cases, protect neurons from damage. Since neurons of higher perceptive, emotional, homeostatic and motor centers (neocortex, hippocampus, hypothalamus and cerebellum) are especially vulnerable to damage, it is tempting to assume that the sensitivity of higher nervous centers to injurious factors is not only an annoying occurrence, but rather plays an essential role for the fulfillment of higher neural functions. If this is indeed the case, motivation and reward are related to the transient damage to and recovery of specific brain neurons. In this work we analyze (both experimentally and theoretically) neuronal mechanisms of decisionmaking during sudden changes in the environment and study one of the main elementary form of animal behavior: instrumental conditioning, where there is a specific association between the animal s action and the unconditioned stimulus. Actually, under instrumental conditioning the animal must found correlation between an action and the stimulus which leads to reward or to punishment. Since at the early stage of the learning the animal makes mistakes, instrumental conditioning is also called trial-and-error learning. Investigation of this process on neuronal level

2 may help to understand, if an animal chooses a presumed form of behavior on the basis of statistical evaluation of its experience, or it believes in its previous successive trials. In this paper we show that the advent of instrumental reactions results from the recovery of a neuron after-excitotoxic damage during the motivational excitation, and an action potential in a single neuron can serve as an elemental instrumental reaction at the cellular level. So, motivational-like behavior can be observed in a single cell. Although neuronal reactions are not repeated from trial to trial, a neuron is taught to evaluate the most preferable consequence of its action and to change its excitability (comprehensive review of the literature and detailed description of the original experiments can be found in [1]). This study allows us to propose new model of an artificial neuron: motivational neuron, which can be served as an elementary unit of brain of the robot, which will behave as an artificial animal that feel drives and love it work. 2 Behavior of real neuron An animal s arbitrary actions are aimed at avoidance of punishment or winning of a reward from the environment. In experiment on a single neuron, the animal receives a painful stimulus (punishment) if the trained neuron fails to generate an action potential in response to specific (tactile) stimulus, while nothing happens after any response to another (discriminated) stimulus. Moreover, receiving of the punishment depends on response of the trained neuron solely, while reactions of the other neurons ( control neurons ) does t correlate with the punishment. When choosing an action, an animal evidently uses its previously accumulated experience. However, action generation depends also on the animal s attitude to the intended result. Force acting within an animal to initiate behavior is called drive or motivation. Motivation is a phenomenon consisting of the generation of actions that lead (following interaction with the environment) to the primary goal, which is attainment of a certain optimal state. The experiments were carried out in a semi-intact preparation of non-anesthetized snails (Helix lucorum L.), as it is described in [1]. The basic schedule of the instrumental conditioning was delivered to only a single target neuron in order to ensure that the instrumental reaction occurred within the recorded neuron. The snail received negative reinforcement when the trained neuron did not generate an action potential (AP ) in response to a conditioned stimulus within seconds. The appearance of an unconditioned stimulus did not depend on the generation or failure of a spike in the control neuron or on the presence of a discriminated stimulus (signals, which differ from the conditional stimulus). Ways to learning. An animal must decide: 1) whether or not something important happened in the environment (if it is absent, we have habituation); 2) if there is a correlation between a signal and a punishment or reward (if it is absent, we have pseudoconditioning); 3) if something depends on the animal s own actions and 4) which animal s action leads to the punishment or reward. At each stage the animal has only a fuzzy guide to forthcoming events, but gathers knowledge during learning. Evidently, an animal does not re-evaluate the expectation of a reward/punishment each time but follows its decision for several trials in succession. It evidently hopes to receive a certain result if it generates an instrumental reaction. Therefore, comparison of the cases in which the unconditioned stimulus appeared either after generation or after failure of the AP in the given neuron is the criterion of its AP participation in the instrumental reaction. Absence of a correlation between these events means, that a given neuron does not participate in the instrumental paradigm. An absence of the correlation for the whole brain means that the paradigm is not instrumental at all. In our experiments, trained neuron participation in the instrumental reaction was predetermined by the conditions of the experiment, while the significance of its participation in an instrumental reaction increased with the data accumulation (see Figure 1. After trials 7-10, the trained neuron acquired sufficient knowledge for a conclusion about its AP participation in the instrumental reaction. Nevertheless, it still did not form a correct reaction to this moment of training. It was unknown a priori which reaction, AP generation or AP failure, would play a role in the instrumental action. Although AP failure in the trained and control neuron did not appear at the same time, the control neuron sometimes produced an AP during the same trials in which the trained neuron produced an AP and received information as if its reaction was essential for the instrumental reaction. Up to trials 15-17, both the trained and control neurons decreased their reaction to the conditioned stimulus (Fig. 1). In the middle of training, when the control neuron effectively generated erroneous instrumental reaction, it collected almost significant data against its participation in the conditioning. During the last trials both the trained and the control neuron confidently formed adequate reaction. So, we may conclude that the system acquired information relative to instrumental conditioning, through several complex stages with varying intermediate conclusions.

3 The important feature of conditioning is the change in the membrane potential during training session. A decrease of a membrane s potential by a motivational excitation can induce neuronal damage, alternate ion homeostasis and disturb excitability. Unconditioned stimuli depolarized neurons [1], but shift of the membrane potential, evidently, can be compensated by some homeostatic processes displaying itself after a break in conditioning for 5-10 minutes before the beginning of extinction. Our data revealed that the conventional properties of a neuron, such as membrane potential maintenance and spike generation, were disturbed during instrumental learning. The trained neuron depolarized during developing of a motivation. Formation of the local instrumental reactions (spike generation) at the end of elaboration was accompanied by a recovery of the membrane potential. The origination of both true and erroneous instrumental reactions was observed during a transient increase in the membrane potential and the rise of neuronal instability in the corresponding neuron. Therefor, depolarization is not immediate cause of the AP generation while hyperpolarization is not the immediate cause of the AP failure. It is reasonable to suppose that the motivational excitation is induced by excitotoxic damage in the trained neuron. Compensational hyperpolarization decreased the damage and recovered spike generation. A similar phenomenon was found in the motor neuron of the mollusk Aplysia after the delivery of strong sensitizing stimuli. Motivational behavior is present at the neuronal level and furthermore homeostasis of a neuron is evidently a unit of motivational behavior. A neuron is a specialized cell for behavior control. Therefor, why must a neuron be deprived of the capability of feeling primitive sensations that non-differentiated cells possess? If a neuron does feel sensation, then what is the demand that the neuron aspires to satisfy? Although a neuron needs to support many important characteristics, only one integral characteristic may affect the sole output of the neuron. This characteristic is the level of cellular excitation, which is intimately connected with the cellular damage/protection. We may suppose that excitation and inhibition are perceived by a neuron as negative and positive sensations. In fact, treatments that protect neurons usually inhibit neurons and exert psychotropic actions related to relief, while the treatments, that induce damage, excite neuron and intensify motivations. In above mentioned experiments it has been demonstrated that AP in a single neuron can serve as an elemental instrumental reaction at the cellular level. Therefore, motivational behavior can be studied at the neuronal level, just as motivational-like behav- Figure 1: Behavior of neuronal activity (experimental data through n=46 neurons) during acquisition of instrumental conditioning. A - trained neuron, B - control neuron. Bars show variation of the experimental data. ior is observed in a single cell. Motivational excitation induces transient neuronal damage, which is conveyed by the shift of some important inner constant: membrane potential decrease, inner calcium concentration growth, etc. Neuronal homeostatic mechanisms compensate for this shift. The compensation is steady if the properties of the environment are simple enough, as during habituation and classical conditioning. Based on the above consideration, we may conclude that the very moment of physiological regulation of damage/protection is also a moment of psychological sensation. Therefor, we may conclude, also, that damage/protection processes in the neurons can be considered as the physiological basis of the motivations. We feel drives because we are mortal and carry out our major needs, such as the aspiration to live and avoidance of injury. Consciousness consists of being between satisfaction and depression that is between life and death, and the ability to feel rests on the fact of mortality. 3 Fuzzy dynamics of neuron Which logic does the brain use for description of an environment? The choice of the means for description of the environment is determined by the primary features of the brain and one of such features is capability of perceiving. So, it seems reasonable to employ such formal notion of perception, which the brain it-

4 self may use for the description of an environment. In this work we want to show that the subjective attitude of the brain to an expected event can be the reason for the advent of the brain s logic of perception (fuzzy logic). Based on the experimental results were presented in the Sec. 2 1 of this work and data from literature we have formulated several rules that characterize a neuronal behavior in a learning process (see details and review of the literature in [1]. Rule 1 Activity of a neuron increases with neuronal damage and decreases with neuronal protection. sub-rule Even if punishment follows after a signal, but expectation of this punishment is low, then a response to this signal can decrease during the nearest time. Rule 2 The damage increases with a punishment or with a high activity of the surrounding neurons and decreases with a reward and non-violent activity of the surrounding neurons. sub-rule Reward is high, if expectation of the punishment is high, but the punishment is absent. Otherwise, reward is small. Rule 3 Expectation of punishment increases if corresponding correlation between AP generation and punishment is high and decreases otherwise. Rule 4 Compensation is turned on either after an AP generation or after AP absence depending upon for which one of the cases the expectation of punishment is higher. If these expectations are close, then compensation is turned on for both types of the neuronal response. These rules allow us to develop a mathematical theory of an adaptation of single neuron by representing of a neuron behavior as trajectory in a 7D state space: S = {x, Θ}: x 1 (t) relative value of AP activity ± x 2 (t) relative value of the protection/damage ± x 3 (t) relative value of the protection/damage compensation Θ ν (t) relative value of the environment reactions expectation (ν = a, b, c, d) where an environment is described by the variable r, which is determined by environmental reactions: punishment, reward or absence of the reaction. In our case, r can be represented as a function of AP activity and expectation of a punishment: r = r(x 1, Θ ν ), (1) where negative r corresponds to the punishment and the positive one corresponds to the reward. The Rules 1-4 are expressed as natural language sentences, rather than as precise mathematically looking laws. One of the main causes of such form of representation of inferences from an experimental data is quite general. In a real experiment only a few parameters of a system are observable and controllable, while a number of the other ones are hidden or remain out of control. For a complex phenomenon it leads to instability in experimental results from trial to trial and makes it difficult to estimate accuracy of the obtained values. In fact, for such a phenomenon fair description of a system behavior is based on our percept of observed tendencies rather than on precise numerical values of the experimental data. The second cause is specific for the considered system. Since a neuron is an elementary unit of processing of perceptions, such structure of the rules directly reflects biological meaning of the neuronal functioning. So, if we want to develop an adequate mathematical theory of a neuron adaptation, we, are in a some sense, compelled to search an apparatus, which could directly operate with perceptions as with mathematical objects. Two decades ago L.Zadeh proposed the theory, which is intended to deal with problems expressed in terms of perceptions. The general concept of this theory has been used for developing of Fuzzy Dynamics - theory of a system evolution, which directly operates with qualitative terms of common human language. It has been shown early that basic dynamics equation of this theory has of the form (see [1] and references there): µ(x, t + δ) = sup v T {Γ(v, x, t); µ(x vδ, t)}. (2) where function µ(x, t) has meaning of the possibility that system is in the point x at the time t, Γ(v, x, t) - is the possibility that system has velocity v in the point x at the time and T {Γ; µ} is representation of the logical connective (Γ µ). For continuous time evolution the Eq.(2) takes of the Liouville-like form: µ t = n V n (x, µ, t, µ) µ x n. (3) with informational velocity - V is obtained as: Γ(V, x, t) µ = λ ; ; T {Γ(V, x, t); µ} = µ. (4) V n The function Γ(v, x, t) is derived directly from the rules 1-4. In order to do this, we should introduce linguistic variables Positive, Negative, etc.. Then, symbolizing them as: P - N - PL - S - NL - Positive, Negative, Positive Large, Small, Negative Large,

5 we obtain symbolic of the rules 1-4, which are represented in the Table??. Here x, Θ corresponds to the trained neuron, y, Θ to the surrounding ones (control neurons in our experiments). The quantities v i and v α designate velocities of changing of the variables x i and Θ α. 1.1 v 1 is P x 2 is NL 1.2 v 1 is N x 2 is PL [r is NL Θ b is S] 2.1 v 2 is P (r is PL y 1 is S) (x 2 is N x 3 is N) } ) 2.2 v 2 is N [x 2 is P x 3 is P] r is NL y 1 is PL 3.1 v 3 is P x 2 is PL [x 3 is N x 2 is S] 3.2 v 3 is N (x 3 is P x 2 is S) ([x 2 is NL] { [x 1 is PL Θ a Θ b is PL] [Θ a Θ b is S] [x 1 is S Θ a Θ b is NL] } ) a.1 v a is P x 1 is PL r is NL a.2 v a is N [x 1 is PL (r is S r is PL)] b.1 v b is P x 1 is S r is NL b.2 v b is N [x 1 is S (r is S r is PL)] c.1 v c is P x 1 is PL r is S c.2 v c is N [x 1 is PL (r is NL r is PL)] d.1 v d is P x 1 is S r is S d.2 v d is N [x 1 is S (r is NL r is PL)] In according the sub-rules 1,2 an environmental response - r for the trained neuron is defined as r is NL x 1 is S r is PL x 1 is PL Θ a is PL r is S x 1 is PL Θ a is S while for the surrounding (control) neurons: r is NL x 1 is S r is PL x 1 is PL [(Θ a is PL) (Θ b is PL)] r is S x 1 is PL [Θ a is S Θ b is S] Because sub-rules n.1 - n.2 are related by connective OR, while the rules are related by connective AND, symbolic form of the Γ is: Γ = ( 1.1 ) ( ) ( ) 3.2 ( a.1 ) ( a.2 b.1 ) b.2 (5) ( c.1 ) ( c.2 d.1 ) d.2. (6) The linguistic variables can be represented as membership functions P (z), N(z), which is truth values of the expressions: z is positive, z is negative, etc. Note, that all these functions can be expressed in term of the one function, say, P (z): N(z) = P ( z), P L(z) = P (z a), NL(z) = P ( z a), S(z) min{1 P L(z); 1 NL(z)}. (7) By using (7) and representation of the logical connectives we can find the truth value of each rule in the Table??. For example, the truth value of the first rule is: T 1.1 = T {P (γ 1 v 1 ); P ( x 2 a 2 )}, where coefficient γ n reflects a time-scale of dynamics of the corresponding variable. Finally, the function Γ(v, x, Θ) is obtained as : where Γ(v, x, Θ) = χ(v, x, Θ) sup v χ(v, x, Θ), (8) χ(v, x, Θ) = T {max [T 1.1 ; T 1.2 ] ; max [T 2.1 ; T 2.2 ] ; max [T 3.1 ; T 3.2 ] ; max [T a.1 ; T a.2 ] ; max [T b.1 ; T b.2 ] ; max [T c.1 ; T c.2 ] ; max [T d.1 ; T d.2 ]}. (9) Solution of Eq. (3) with V n from (4) is shown on Fig. 2. Comparison Figure 2(A,C) with Figure 1(B,A) shows that agreement between theory and the experiment is very good. It should be noted that sudden changes in neuron behavior, which are seen in the experimental data and are predicted by the fuzzy dynamics, are fundamental ones and it is straight consequence of the representation of the logical connective OR as (A B) max{µ A ; µ B }. It should be emphasized that such a representation is not result of our free choice, but it is dictated by the neuron s physical features (see [1] for details). Sudden alterations in the neuron s behavior have an effect on macro-behavior of an animal. It is well known that even knowing a good solution to a given problem, an animal from time-to-time tries to find a new solution and if the new solution is a worse one the animal returns to the previous behavior. Such researcher s instinct is very beneficial, since it enables the animal continuously optimize its behavior in changing environmental conditions. Note, that in the above presented approach such behavior is neither the consequence of random inner influences of the neural system and nor only the result of the sudden changes in environment, but rather a fundamental feature of neurons. It seems very likely, that the inner logic of the neuron s behavior is close to fuzzy logic. The idea to use some of the features of physiological processes in cybernetics is very old, but it is still

6 References: [1] U. Sandler, L. Tsitolovsky, The Attitude to Expected Reward as the Basis for Fuzzy Logic of the Brain: We Feel Drives Because We are Mortal (Invited Paper). Int.J.Comput.Cogn., 3, 2005, pp Figure 2: Neuron learning under instrumental conditioning conditioning (theory). Bars shown regions of the most preferable reaction of the neurons have been predicted by the theory. A - activity, B-damage/protection, E - expectation [Activity P unishment], F - expectation [Absence activity P unishment] for the control neuron. C - activity, D - damage/protection, G - expectation [Activity P unishment], H - expectation [Absence activity P unishment] for the trained neuron attractive. Our approach makes it feasible to design new kinds of artificial neurons: motivational artificial neurons, because Fuzzy Rules in the Table are naturally computerized both on the software and hardware levels. Motivational artificial neurons seem to be very promising as basic elements of the brain of the feeling robot. Behavior of such robots will be initiated by a general defensive motivation, which is expressed as a few artificial drives like energy recuperation, avoidance of injury and aspiration to survive, while the robots main task can be implemented as an analog of an animal s sexual motivation. As most of the fuzzy systems, motivational artificial neurons are easily hybridized with almost any sensors, actions controllers and long-term memory system. So, in a feeling robot performance of the main task, aspiration to survive, trial-and-error learning and instinct of researcher will be naturally combined. If a robot have to act autonomously in a poorly predictable and a hard environmental condition, a motivational paradigm may be significantly more effective than conventional reinforcement learning approaches. Moreover, preliminary simulations shown that process of decision-making process of the motivational artificial neurons is quite fast and does t require unreasonable computational resources.