Influence of Neural Delay in Sensorimotor Systems on the Control Performance and Mechanism in Bicycle Riding

Transcription

1 Neural Information Processing Letters an Reviews Vol. 12, Nos. 1-3, January-March 28 Influence of Neural Delay in Sensorimotor Systems on the Control Performance an Mechanism in Bicycle Riing Yusuke Azuma an Akira Hirose Department of Electronic Engineering, The University of Tokyo Hongo, Bunkyo-ku, Tokyo, , Japan (Submitte on December 1, 27) Abstract Neural sensorimotor systems have unavoiable elay in the processing. In this paper, we investigate the influence of the elay on control performance in numerical experiment with simple networks. Obtaine results suggest not only the fact that the elay egraes control performance of reflex motion in proportion to the amount of the elay, but also the possibility that it influences the signal generation of relevant volitional motion. The motion generation mechanism seems epenent on the amount of elay. That is, when the elay increases beyon a certain threshol, the control strategy changes from a feeback metho into a preictive one. An appropriate elay can even be in favor of the volitional motion learning in total. Keywors Motion learning, Developmental learning, Reinforcement learning, Reflex motion, Volitional motion 1. Introuction Biological sensorimotor systems have unavoiable elay when sensory signals yiel motor effects because of the transmission an processing retaration as well as the mechanical inertia of objects. In the human beings, the amount of the elay is typically several tens to a few hunres of millisecons. When we nee a quicker response, we nee to overcome this elay. In this sense, we have to be aaptive in a preictive manner [1]. An interesting experiment was presente in a paper by Ishia & Sawaa [2]. A test subject tracks a moving target on a computer screen with a mouse (cursor) for various perioic motions. Then there is a region of the perio of the reciprocating motion of the target where the cursor precees the target an, in this case, the manner of the preceing is such that a transient locus error is minimize when the target moves abruptly non-perioically. The fact suggests, at least, that the human being behave in a certain special way to overcome the unavoiable elay. In this paper, we investigate the elay effect on simple sensorimotor neural networks experimentally. We assume a task to rie a bicycle to observe reflex an volitional motions, in which a feeforwar network is loae with various amount of elay. We fin in the experiments that the elay egraes the reflex motion performance in proportion to the elay time up to a certain elay amount. For a larger elay, the performance is rapily eteriorate. In aition, the elay influences the performance in learning volitional signal generation to realize a motion relevant to the previously learnt reflex motion. It is also suggeste that a moerate elay sometimes results in a better performance of the learning of the volitional motion in total because the control strategy changes from a feeback metho into a preictive one. 2.1 Bicycle an human boy 2. Moelling Figure 1 is a top view of a bicycle moving in a small time step t. The avance s is relate to the istance between front an rear wheels, hanle angle, curvature raius of bicycle trajectory R, an change in 43

2 Influence of Neural Delay in Sensorymotor Systems Yusuke Azuma an Akira Hirose ψ s Centrifugal force Bicycle β l Human boy Center of gravity Force of gravity Figure 1. Top view an rear view of a man riing bicycle. the forwar movement irection ψ as Since R = s/ ψ = / sin, we obtain s sin = sin ψ ψ (1) ψ = v t/r (2) Figure 1 is a rear view of the bicycle an human boy, which is analogous to a ouble inverte penulum, where an β enote the tilt angle of bicycle to the groun an that of human to the bicycle, respectively, m is total weight of the bicycle an the human boy, an l is the length from the groun contact point of the wheel to the center of gravity of the total mass. Since the rate of change in the angular momentum ml 2 β is equal to the torque, that is a ifference between the force of gravity mg an centrifugal force mv 2 /R, wehave ml 2 β = mgl sin β lmv 2 cos β/r (3) We calculate the state evolution by (3). A falling is expresse as β = π/2. The values of the physical parameters was chosen as =1.m, both of the heights of the bicycle an the boy.8m, boy weight 7kg, bicycle weight 1[kg], bicycle velocity constantly 3m/s for simplicity, an the calculation time step t=1ms. 2.2 Neural network We moel reflex motion, which is quick an simple, an volitional motion, which may be slow an complex. When we ri a bicycle, we control our sitting posture an hanle irection to balance ourselves. If we feel falling to the right, for example, we quickly rive the hanle to the right an tilt our boy to the left. In the simulation presente below, we control the hanle an the boy using a network. The bicycle velocity is kept constant for simplicity. We realize a reflex control by a simple single-layer feeforwar neural network. By choosing neutral position of boy an irection of hanle as, we fin spatial symmetry with respect to position or irection. Therefore, we employ an activation function of hyperbolic tangent with which we represent left an right by plus an minus numbers. Figure 2 shows the network constructions we employ in this paper to observe the influence of elay. First, as a preliminary experiment, we conuct a reflex motion experiment using Network with a single input of bicycle tilt angle. Neural elay is expresse as. Next, using Network, we examine a reflex motion using both an its changing rate (time erivative). Finally, we observe a relate volitional motion using Network (c) or () with aitional inputs of volitional signals fe to the boy tilt neuron a turn an hanle angle neuron a turn. 44

3 Neural Information Processing Letters an Reviews Vol. 12, Nos. 1-3, January-March 28 w w w w w w aturn aturn w w w w w w w w aturn (c) () Figure 2: Neural network constructions for reflex motion simulation with one input of tilt angle, that with two inputs of tilt angle an its erivative, (c)volitional motion with a volitional signal for boy tilt neuron a turn only, an ()that with volitional signals a turn an another for hanle angle neuron a turn in aition. Neural elay is for all of the neurons. Time erivative of bicycle tilt [ra/s] Bicycle tilt [ra] Bicycle tilt [ra] Figure 3. State trajectories of single-input system for elay of = an =5ms. [ra/s] Time erivative of bicycle tilt 3. Reflex Motion in Bicycle Riing 3.1 Single input network (Preliminary experiment) We examine the elay influence on reflex motion for Network with a single input of bicycle tilt only. We generate neural weights at ranom an observe the behavior of the sensorimotor system. The initial state is =.1ra an =.1ra/s. Typical results are illustrate in Fig.3 as state trajectory in the space. Figure 3 is a result when the network has no elay. We fin a stable trajectory. On the contrary, Fig.3 shows a typical result when a small elay (5ms) exists. The trajectory is unstable, an the bicycle finally falls. It is obvious that even a small elay influences the behavior of a sensorimotor system. 3.2 Two input network (Bicycle tilt an its changing rate) Human beings also catch outer worl motion in the vision. We simplify the fact into an availability of time erivative information of tilt,, in aition to itself. The network construction is Fig.2. 45

4 Influence of Neural Delay in Sensorymotor Systems Yusuke Azuma an Akira Hirose (c) Figure 4: Neural weight maps w w an w w for trials resulting in stabilization when the elay is =, = 1ms, an (c) = 2ms, respectively. Again we choose neural weights at ranom to examine if the bicycle fluctuation iminishes or not, where iminish means that both an remains within ±1 3 ra for more than 5s. In Fig.4, we plot the weights w, w, w, an w only when the fluctuation iminishes, i.e., when the rie is successful. The weights construct four-imensional space in total, while the chart represents them on the two planes. The area where the points exist shows the parameter region of stable control. Figure 4 shows the result when there is no elay. The stable area is wie. Figure 4 is a result for a elay 46

5 Neural Information Processing Letters an Reviews Vol. 12, Nos. 1-3, January-March 28 Tilt erivative [ra/s] Tilt [ra] Figure 5. Stable state trajectory when the time erivative tilt angle is available as an input in reflex motion Volitional signal aturn Time t[s] y[m] x[m] Figure 6: Example of volitional signal input to the boy tilt neuron a turn an obtaine right-turning locus in a right turn task..4 Time erivative of. bicycle tilt angle [ra/s] Bicycle tilt angle [ra] Figure 7. State trajectory obtaine for the right turn behavior shown in Fig.6. of 1ms. The stable region is limite in a belt. Figure 4(c) is for a elay of 2ms. The belt is much thinner. For a larger elay, we cannot fin out stable regions any more. A larger elay obviously restricts the stable parameter area. An example of stable state trajectories is shown in Fig.5. 47

6 Influence of Neural Delay in Sensorymotor Systems Yusuke Azuma an Akira Hirose Bicycle tilt [ra].1 Stabilization time T Peak level less than 1% Stabilization time T[s] Time t[s] Delay time [ms] Figure 8: Definition of stabilization time T in the waveform of bicycle tilt versus time t, an obtaine stabilization time T versus elay time after completion of reinforcement learning of four weights w, w, w an w. 4. Volitional Motion Relevant to Reflex Motion Learnt Previously 4.1 Preliminary experiment to observe the relationship between reflex motion an volitional motion to turn Action to turn is a volitional motion with consciousness, which is completely ifferent from a reflex motion. However, in the case of a volitional motion that is relevant to a reflex motion, we assume that the reflex also works simultaneously because of its reflex nature. In other wors, we eal with a turn motion not as a completely inepenent volitional motion but, instea, a mixture generate with a volitional control signal an a previously learnt reflex signal. A possible neural moel is illustrate in Fig.2(c). As a preliminary experiment, we conuct an experiment to see what motion the network in Fig.2(c) generates when it receives a simple input signal shown in Fig.6. We assume a elay of =1ms. Figure 6 presents the resulting locus. When it receives the signal a turn, it starts to turn, an when the signal ens, it also stops to turn, without falling own. Figure 7 shows the state trajectory. The state starts from the center of the left curl at the coorinate origin. When the volitional signal is input, the state migrates to another curl on the right-han sie, an when the signal ens, it returns to the initial state at the origin. The volitional signal shifts the initial stable state to another stable one, where the boy is slightly askew, resulting in a smooth right turn. This motion is generate cooperatively by the reflex motion learnt in Section 3.2. In an analogous way, when we applie another volitional signal a turn, which moulates the hanle angle,a similar reaction was observe. In reality, we can expect that both of a turn an a turn together generate a smooth turn motion. The network is shown in Fig.2(). 4.2 Volitional Task, conitions, an preparatory reflex motion learning We investigate the elay influence on the learning of volitional behavior to turn to the right. We employ Network (), shown in Fig.2(), having aitional inputs of volitional signals at boy tilt an hanle angle neurons, a turn an a turn, respectively. The learning process is as follows. First, we conuct a preparatory experiment of reinforcement learning of reflex motion only for bicycle stabilization. It simulates unconscious aily learning to acquire the reflex reaction. In the learning, the four weights, w, w, w an w, are ajuste to minimize the stabilization time T efine in Fig.8. This process optimizes the reflex motion. Figure 8 shows the stabilization time T versus neural elay time. We fin that T grows longer in proportion to up to 25ms. For a larger, the stabilization time T rastically increases. The elay of 25ms is foun to be a critical value in the present system. After this reflex motion learning is complete, we go on to the following volitional motion learning where the system learns the volitional signal waveforms, while the reflex weights are fixe in this stage. The simulation scheule is similar to that in evelopmental learning [3]. Figure 9 illustrates the task in the volitional motion to 48

7 Neural Information Processing Letters an Reviews Vol. 12, Nos. 1-3, January-March 28 4[m] Ieal path 4[m] Actual locus Locus error e Volitional signals aturn (t) an aturn (t) aturn (t) aturn (t) Aturn Aturn Cosine shape waveforms y x Starting point Start time ts En time te Figure 9: The locus error e to be accumulate an average in the task to turn to the right, an waveform of the volitional signals fe to boy tilt neuron a turn an hanle angle neuron a turn. The amplitue values, A turn an A turn, an their common start time t s, an en time t e are optimize through reinforcement learning. Turn signal start time ts an en time te[s] En time te Start time ts Delay time [ms] Figure 1. Turn signal start time t s an en time t e versus elay time. t turn to the right. The eviation of the actual locus from the ieal path is accumulate step by step to yiel average locus error e, which is evaluate in the reinforcement learning. To change the boy tilt an the hanle irection, we fee volitional signals a turn (t) an a turn (t) to the respective neurons, which are variable an optimize in the volitional-motion learning. Figure 9 shows the waveforms of the signals a turn (t) an a turn (t), which correspon to a half cycle of cosine waveform, with four parameters of start time t s, en time t e, signe (positive or negative) amplitue for boy A turn, an that for hanle A turn. The volitional signals are optimize in terms of these four parameters. That is, the system learns optimal t s, t e, A turn, an A turn in another stage of reinforcement learning. 4.3 Results of reinforcement learning Figure 1 shows the plots of start time t s an en time t e of volitional signal injection, which is optimize in reinforcement learning for various neural elay time. Each elay value contains several successful learning results corresponing to multiple trials. The origin of the time is the time when the bicycle starts the starting point in Fig.9. When is smaller than 1ms, the ispersion of t s an t e is small. The signal uration (t e t s ) is foun minimum at 12ms. For a larger, the ispersion graually grows larger. When > 25ms, the ispersion gets infinitely large, an the learning process often fails. Figure 11 shows the average locus error E versus the neural elay. We fin that E is minimum at 12ms. Figure 12 shows the obtaine signe amplitue of the volitional signals, A turn an A turn, versus the elay time. Their absolute values reuces in accorance with the increase of the elay time. 49

8 Influence of Neural Delay in Sensorymotor Systems Yusuke Azuma an Akira Hirose Average locus error E[m] Delay time [ms] Figure 11. Average locus error E versus elay time. Turn signal amplitue Aturn an Aturn Aturn Aturn Delay time [ms] Figure 12. Optimize signe amplitues of volitional turn signals A turn an A turn versus elay time. 4.4 Discussion In Fig.1, the volitional signal uration is minimum at aroun =12ms. This value agrees with the minimum error elay in Fig.11. It is interesting that a moerate elay results in a better learning of volitional motion rather than a small (< 5ms). In Fig.12, for a larger, the absolute values of the signe amplitues A turn an A turn converges at. The reason lies in the slow feeback in reflex motion for a large. That is, large volitional signals make the bicycle fall own because of the slow feeback. They reuce exponentially to the change in, while the error E changes only moerately. In the above experiment, we have foun that a moerate elay leas to a better learning in the volitional motion to turn to the right in the bicycle riing. An explanation is given as follows. When the neural elay is small, the sensorimotor system observes the state instantaneously without elay, an controls the bicycle in a feeback manner by spening a certain time. Contrarily, when is large, the system cannot observe the effect of its control an, therefore, it cannot employ a feeback strategy. Instea, the system generates signal in a cavalier fashion, or in a preicting manner, an then just waits for the result. Another explanation may be available, if we regar the learning as a class of search, as follows. In Fig.4, we foun that the successful parameter region becomes narrower when is larger in the reflex motion learning. If the elay is small, the reflex learning may stop cursorily before it falls a better state because the task is too easy. In contrast, if a moerate elay exists, the reflex motion learning makes progress further to a much better state. It may lea to a better learning of volitional motion. A too large elay (>2ms) results in fatal ifficulty in total. 5

9 Neural Information Processing Letters an Reviews Vol. 12, Nos. 1-3, January-March Conclusion A sensorimotor system learns how to overcome unavoiable elay to respon to environment. It has been shown that a moerate elay sometimes results in a better learning of relevant volitional motion. An explanation is as follows. When the neural elay is small, the sensorimotor system observes the state instantaneously an continuously, an controls the bicycle in a feeback manner. Contrarily, when the elay is moerately large, the system cannot observe the effect of its control output an, therefore, it cannot employ a feeback strategy. Instea, the system generates signal in a cavalier fashion, or in a preicting manner, an then just waits for the future result. It has been suggeste that the amount of elay influences the neural learning strategy. References [1] R. Miall, D. Weir, an J. Stein, Manual tracking of visual targets by traine monkeys, Behavioural Brain Research, vol.2, no.2, pp , May [2] F. Ishia an Y. Sawaa, Human han moves proactively to external stimulus: An evolutional strategy for minimizing transient error, Physical Review Letters, vol.93, no.16, p.16815, October 24. [3] A. Hirose, Y. Asano, an T. Hamano, Developmental learning with behavioral moe tuning by carrierfrequency moulation in coherent neural networks, IEEE Transactions on Neural Networks, vol.17, no.6, pp , November 26. Yusuke Azuma receive the B.E. an M.E. egrees both in electronic engineering from the University of Tokyo, Tokyo, Japan, in 25 an 27, respectively. He is currently with Elysium Inc. His research interest inclues aaptive control an computer aie esign. Akira Hirose receive the Ph.D. egree in electrical engineering from the University of Tokyo, Tokyo, Japan, in In 1987, he joine the Research Center for Avance Science an Technology (RCAST), the University of Tokyo, as a Research Associate, where he was engage in research on optical communications an measurement. In 1991, he was appointe an Instructor at the RCAST, an starte neural network research. From 1993 to 1995, on leave of absence from the University of Tokyo, he was with the Institute for Neuroinformatics, University of Bonn, Bonn, Germany. He became an Associate Professor at the RCAST in 1995, an a Professor at the Department of Electronic Engineering, The University of Tokyo, in 27. The main fiels of his current research interest are wireless electronics an neural networks. Dr. Hirose is a member of the IEEE an the JNNS. 51