Mirror neurons and imitation: A computationally guided review

Transcription

1 Neural Networks 19 (2006) Special issue Mirror neurons and imitation: A computationally guided review Erhan Oztop a,b, *, Mitsuo Kawato a,b, Michael Arbib c a JST-ICORP Computational Brain Project, Kyoto, Japan b ATR, Computational Neuroscience Laboratories, Hikaridai Seika-cho, Soraku-gun, Kyoto , Japan c Computer Science, Neuroscience and USC Brain Project, University of Southern California, Los Angeles, CA , USA Abstract Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals the presence of mirror systems (not individual neurons) in the human. Current conceptual models attribute high level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often overlook the computational requirements for posited functions, while too many computational models adopt the erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines the gap between conceptual and computational models and points out the research effort required from both sides to reduce this gap. q 2006 Elsevier Ltd. All rights reserved. Keywords: Mirror neuron; Action understanding; Imitation; Language; Computational model 1. Introduction Many neurons in the ventral premotor area F5 in macaque monkeys show activity in correlation with the grasp 1 type being executed (Rizzolatti, 1988). A subpopulation of these neurons, the mirror neurons (MNs), exhibit multi-modal properties responding to the observation of goal directed movements performed by another monkey or an experimenter (e.g. precision or power grasping) for grasps more or less congruent with those associated with the motor activity of the neuron (Gallese, Fadiga, Fogassi, & Rizzolatti, 1996; Rizzolatti, Fadiga, Gallese, & Fogassi, 1996). The same area includes auditory mirror neurons (Kohler et al., 2002) that respond not only to the view but also to the sound of actions with typical sounds (e.g. breaking a peanut, tearing paper). The actions associated with mirror neurons in the monkey seem * Corresponding author. Address: Department of Cognitive Neuroscience, ATR, Computational Neuroscience Laboratories, Hikaridai Seika-cho, Soraku-gun, Kyoto , Japan. Tel.: C ; fax: C address: erhan@atr.jp (E. Oztop). 1 We restrict our discussion to hand-related neurons; F5 contains mouthrelated neurons as well /$ - see front matter q 2006 Elsevier Ltd. All rights reserved. doi: /j.neunet to be transitive, i.e. to involve action upon an object and apply even to an object just recently hidden from view (e.g. Umilta et al., 2001). It is not possible to find individual mirror neurons in humans since electrophysiology is only possible in very rare cases and at specific brain sites in humans. Therefore, one usually talks about a mirror region or a mirror system for grasping identified by brain imaging (PET, fmri, MEG, etc.). Other regions of the brain may support mirror systems for other classes of actions. An increasing number of human brain mapping studies now refer to a mirror system (although not all are conclusive). Collectively these data indicate that action observation activates certain regions involved in the execution of actions of the same class. However, in contrast to monkeys, intransitive actions have also been shown to activate motor regions in humans. The existence of a (transitive and intransitive) mirror system in the human brain has also been supported by behavioral experiments illustrating the so-called motor interference effect where observation of a movement degrades the performance of a concurrently executed incongruent movement (Brass, Bekkering, Wohlschlager, & Prinz, 2000; Kilner, Paulignan, & Blakemore, 2003; see also Sauser & Billard, this issue, for functional models addressing this phenomenon). Because of the overlapping neural substrate for action execution and observation in humans as well as other primates, many researchers have attributed high level cognitive

2 E. Oztop et al. / Neural Networks 19 (2006) functions to MNs such as imitation (e.g. Carr, Iacoboni, Dubeau, Mazziotta, & Lenzi, 2003; Miall, 2003), action understanding (e.g. Umilta et al., 2001), intention attribution (Iacoboni et al., 2005) and on the finding of a mirror system for grasping in or near human Broca s area (evolution of) language (Rizzolatti & Arbib, 1998). 2 We stress that, although statements are often made about mirror neurons in humans, we have data only on what might be called mirror systems in humans connected regions that are active in imaging studies both when the subject observes an action from some set and executes an action from that set, but not during an appropriate set of control tasks. Brain imaging results show that mirror regions in human may be associated with imitation and language (Carr et al., 2003; Fadiga, Craighero, Buccino, & Rizzolatti, 2002; Iacoboni et al., 1999; Skipper, Nusbaum, & Small, 2005), but there are no corresponding data on mirror neurons. Moreover, monkeys do not imitate (but see below) or learn language and so any account of the role of mirror neurons in imitation and language must include an account of the evolution of the human mirror system (Rizzolatti & Arbib, 1998) or at least the biological triggers that can unleash in monkeys a rudimentary imitation capability that goes beyond those they normally exhibit, though still being quite limited compared to those of humans (Kumashiro et al., 2003). We thus argue that imitation and language are not inherent in a macaque-like mirror system but instead depend on the embedding of circuitry homologous to that of the macaque in more extended systems within the human brain. A general pitfall in conceptual modeling is that an innocent looking phrase thrown in the description may render the model implausible or trivial from a computational perspective, hiding the real difficulty of the problem. For example, terms like direct matching and resonance are used as if they were atomic processes that allow one to build hypotheses about higher cognitive functions of mirror neurons (Gallese, Keysers, & Rizzolatti, 2004; Rizzolatti, Fogassi, & Gallese, 2001). One must explain the cortical mechanisms which support the several processing stages that transform retinal stimulation caused by an action observation into the mirror neuron responses. Another issue is to clarify what is encoded by the mirror neuron activity. Is it the motor command, the meaning or the intention of the observed action? In an attempt to explain the multiplicity of functions attributed to mirror neurons, it has been recently speculated that different set of mirror neurons are involved in different aspects of the observed action (Rizzolatti, 2005). We will review various conceptual models that pay little attention to this crucial reservation, and then review several computational models in some detail: a learning architecture with parametric biases (Tani, Ito, & Sugita, 2004); a genetic algorithm model which develops networks for imitation while 2 Recent reviews of the mirror neuron and mirror system literature are provided by Buccino, Binkofski, and Riggio (2004), Fadiga and Craighero (2004) and Rizzolatti and Craighero (2004). yielding mirror neurons as a byproduct of the evolutionary process (Borenstein & Ruppin, 2005); the mirror neuron system (MNS) model that can learn to mirror via selfobservation of grasp actions (Oztop & Arbib, 2002) and is closely linked to macaque behavior and (somewhat more loosely) neurophysiology; and models which are not restricted in this fashion: the mental state inference (MSI) model that builds on the forward model hypothesis of mirror neurons (Oztop, Wolpert, & Kawato, 2005), the modular selection and identification for control (MOSAIC) model that utilizes multiple predictor controller pairs (Haruno, Wolpert, & Kawato, 2001; Wolpert & Kawato, 1998), and the imitation architecture of Demiris and Hayes (2002) and Demiris and Johnson (2003). 2. Mirror neurons and action understanding Mirror neurons, when initially discovered in macaques, were thought to be involved in action recognition (Fogassi et al., 1992; Gallese et al., 1996; Rizzolatti et al., 1996) though this laid the basis for later work ascribing a role in imitation to the human mirror system. Although the term action understanding was often used, the exact meaning of understanding as used here is not clear. It can range from act according to what you see, to infer the intentions/mental states leading to the observed action. In fact, the neurophysiological data simply show that a mirror neuron fires both when the monkey executes a certain action and when he observes more or less congruent actions. In these experiments, he is given no opportunity to show by his behavior that he understands the action in either of the above senses. Gallese and Goldman (1998) suggested that the purpose of MNs is to enable an organism to detect certain mental states of observed conspecifics via mental simulation. According to this view, mirror neurons could be the precursor of mind-reading ability, being compatible with the simulation theory hypothesis. 3 Again, this involves considerable extrapolation beyond the available data. In particular, mind-reading might involve a quite separate mirror system for facial expression as much as a mirror system for manual actions. Although the suggestion has achieved some positive reception, no details have been provided on how this could be implemented as a computational model. (The MSI model does address this issue, see below). In spite of the computational differences between recognition (and imitation) of facial gestures and recognition (imitation) of hand actions, many cognitive neuroscientists address them both under a generic mirror system. The insula has been found to be a common face-emotion region for both production and understanding of facial gestures and emotions (Carr et al., 2003; Wicker et al., 2003). Although it is tempting to consequently take the insula as formed by instantiating 3 Two predominant accounts of mind-reading exist in the literature. Theory theory asserts that mental states are represented as inferred conjectures of a naive theory whereas according to simulation theory, mental states of others are represented by representing their states in terms of one s own.

3 256 E. Oztop et al. / Neural Networks 19 (2006) an F5-like mirror system for emotion processing, there are crucial differences in how these circuits develop in infancy which shed doubt on the idea of a generic neural mirror mechanism to unify social cognition (Gallese et al., 2004). Manual actions can be compared visually and vocalizations can be compared auditorially the commonality being the matching of an observed action with the output from an internal motor representation through a comparison in the same domain. Unlike hand actions, one s own facial gestures can only be seen with the help of a reflective material or otherwise must be inferred. In fact, infants may learn much about their own facial expressions from the propensity of caregivers to imitate the child. We argue that the learning mechanism of facial imitation is different from hand imitation, which involves learning via social interaction of the kinds we suggested elsewhere (Oztop et al., 2005): You eat A/you have face expression X (visual). I eat A/I feel disgust Y (internal state). Therefore, X (visual) must be Y (feeling of disgust). The elaboration of this mechanism in the brain is beyond the scope of this review. However, the important message here is that manual action understanding and facial emotion understanding pose different problems to the primate brain, which might have found solutions in different organizational principles. 3. Mirror neurons and imitation Learning by imitation is an important part of human motor behavior, which requires complex set of mechanisms (Schaal, Ijspeert, & Billard, 2003). Wolpert, Doya, and Kawato (2003) underline some of those as (i) mapping the sensory variables into corresponding motor variables, (ii) compensation for the physical difference of the imitator from the demonstrator, and (iii) understanding the intention (goal) causing the observed movement. Many cognitive neuroscientists view imitation as mediated by mirror neurons in humans. Although this is a plausible hypothesis, we stress again that this is not within the normal repertoire of the macaque mirror system (but note the discussion below of Kumashiro et al., 2003) and so must if true rest on evolutionary developments in the mirror system. It should also be emphasized that there is a considerable amount of literature addressing imitation without explicit reference to mirror neurons (e.g. see Byrne & Russon, 1998). Since most mirror neurons are found in motor areas it is reasonable to envision a motor control role for mirror neurons. One possibility is that these neurons implement an internal model for control. Current evidence suggests that the central nervous system uses internal models for movement planning, control, and learning (Kawato & Wolpert, 1998; Wolpert & Kawato, 1998). A forward model is one that predicts the sensory consequences of a motor command (Miall & Wolpert, 1996; Wolpert, Ghahramani, & Flanagan, 2001); while an inverse model transforms a desired sensory state into a motor command that can achieve it. The proposal of Arbib and Rizzolatti (1997) that mirror neurons may be involved in inverse modeling plays a central role in recent hypotheses about the neural mechanisms of imitation, which were accelerated by the increasing number of functional brain imaging studies. One recent proposal is that the mirror neurons may provide not only an inverse model but a forward model of the body which can generate action candidates in the superior temporal sulcus (STS) where neurons have been found with selectivity for biological movement (e.g. of arms, whole body) (Carr et al., 2003; Iacoboni et al., 1999). The idea is that STS acts as a comparator that can be used within a search mechanism that finds the mirror neuron-projected action code that matches the observed action best, which is in turn used for subsequent imitation. It is further suggested that the STS-F5 circuit can be run in the reverse direction (inverse modeling) to map the observed action into motor codes (mirror activity) so that a rough motor representation of the observed act becomes available for imitation. According to this hypothesis, the F5- STS circuitry must be capable of producing detailed visual representation of the self-actions. From a computational point of view, if we accept that an observed act can be transformed into motor codes or if we accept the availability of an elaborate motor/visual forward model then imitation becomes trivial. However, the above conceptual model limits imitation to actions already in the observer s repertoire. It is one thing to recognize a familiar action and quite another to see a novel action and consequently add it to one s repertoire. Another concern is that the mirror neurons found with electrophysiological recordings in monkeys are limited to goal directed actions. Indeed, behavioral studies show that chimpanzees cannot handle imitation tasks which do not involve any target objects (Myowa-Yamakoshi & Matsuzawa, 1999). In their efforts to release rudimentary imitation capability in monkeys Kumashiro et al., (2003) used four tests based on objects (cotton-separation, knob-touching, latched box opening and removing the lid from a conical tube) and (with highly variable success) three tests based on facilitating use of a specific effector (tongue-protrusion model, hand-clench and thumbextension) and two based on the movement of a hand relative to the body (hand to nose, hand clap, and hand to ear). Miall (2003) suggested amending the conceptual model of Iacoboni et al. by including the cerebellum. He proposed that the forward and inverse computations required can be carried out by the cerebellum and PPC (posterior parietal cortex). The cerebellum has often been considered the likely candidate for (forward and inverse) internal models (Kawato & Gomi, 1992; Wolpert, Miall, & Kawato, 1998), but an alternative view (related to Miall s) is that cerebellar models act in parallel with models implemented in cerebral cortex, rather than replacing them (Arbib, Erdi, & Szentagothai, 1998). However, the computational problem is still there: how the retinal image is transformed to motor commands in a precise way (inverse problem), and how a precise visual description of one s own body is mentally produced given a motor command (forward problem). For inverse and forward models whose sensory data are limited to the visual domain, the problems are quite severe when the whole body is considered because one cannot completely observe all of one s own body. However,

4 E. Oztop et al. / Neural Networks 19 (2006) observation of the hand in action is possible, enabling forward and inverse model learning. Presumably, the brain does not rely on visual data alone, but integrates it with proprioceptive cues. Although inverse learning is harder than forward learning, in general this does not pose a huge problem, and by using certain invariant representations in the visual domain, the learned internal models can be applied to other individuals movements Mirror neurons and language Rizzolatti and Arbib (1998) built on evidence that macaque F5 is homologous to human Broca s area (an area often associated with speech production) and that human brain imaging reveals a mirror system for grasping in or around Broca s area to propose that brain mechanisms supporting language in humans evolved atop a primitive mirror neuron system similar to that found in monkeys. According to their mirror system hypothesis, human Broca s area s mirror system properties provide the evolutionary basis for language parity (the approximate meaning-equivalence of an utterance for both speaker and hearer). Arbib (2002, 2005) has expanded the hypothesis into seven evolutionary stages: S1: Simple grasping, S2: A mirror system for grasping, S3: A simple imitation system for grasping, S4: A complex imitation system for grasping, S5: Protosign, a hand-based communication system, S6: Protospeech, a vocalization-based communication system, S7: Language, which required little or no biological evolution beyond S6, but which resulted from cultural evolution in Homo sapiens. We do not have space here to comment on all the above steps but one quite distinctive feature of the analysis of stage S2 deserves attention. It asks the question why do mirror neurons exist? and answers because mirror neurons are (originally) involved in motor control. These neurons are located in the premotor cortex, interleaved with other motor neurons, where distal hand movements are controlled. Indeed, one may argue that the visual feedback required for manual dexterity based on observation of the relation of hand to goal object provided mechanisms that were exapted in primate and hominid evolution first for action recognition and then for imitation. Although it is a daunting task to computationally realize this evolutionary model in its complete form (an attempt not without its critics see the commentaries pro and con in Arbib (2005)), the transitions from one stage to the next can be potentially studied from a computational perspective. 4 In general it is not possible to infer the full dynamics from a kinematics observation, but we may assume some approximate solution that yields similar kinematics when applied by the observer. 5. Computational models involving MNs We still lack a systematic neurophysiological study that correlates mirror neuron activity with the kinematics of the monkey or the demonstrator which would allow the computational modeler to test ideas about such correlations. Moreover, if (as we believe) the mirror properties of these neurons as distinct from the conditions which make their acquisition possible are not innate, then the study of the developmental course of these neurons and their function would test computational models of development that could help us better understand the functional role of MNs. A general but wrong assumption in many computational studies of imitation is that mirror neurons are responsible for generating actions (and even sometimes that area F5 is composed only of mirror neurons). Indeed, F5 can be anatomically subdivided into two distinct regions, one containing the mirror neurons, and one containing canonical neurons. The latter are like mirror neurons in their motor properties, but do not respond to action observation. Muscimol injections (muscimol causes reversible neural inactivation) to the part of area F5 that includes mirror neurons do not impair grasping and control ability, but causes only a slowing down of the action (Fogassi et al., 2001). 5 However, when the area that includes the canonical neurons is the target of injection, the hand shaping preceding grasping is impaired and the hand posture is not appropriate for the object size and shape (Fogassi et al., 2001). In the following sections, we will review various computational studies that relate (often at some conceptual remove) to mirror neurons. Unfortunately, most of the modeling is targeted at imitation. Only one, the MNS model of Oztop and Arbib (2002) directly claims to be a model for mirror neurons (although it does not provide computational modules for motor control). 6. A dynamical system approach The first model we review, due to Jun Tani et al. (2004), is aimed at learning, imitation and autonomous behavior generation. The proposed network is a generative learning architecture called recurrent neural network with parametric biases (RNNPB). In this architecture, the spatio-temporal patterns are associated with so-called parametric bias vectors (PB). RNNPB self-organizes the mapping between PBs and the spatio-temporal patterns (behaviors) during the learning phase. From a functional point of view the goal of RNNPB is similar to dynamical movement primitive learning (Ijspeert, Nakanishi, & Schaal, 2003; Schaal, Peters, Nakanishi, & Ijspeert, 2004) in that the behaviors are learned as dynamical systems. However, in the dynamical movement primitive approach, dynamical systems are not constructed from scratch as 5 This may at first appear inconsistent with the view that mirror neurons may assist the feedback control for dexterous movements, but note that the muscimol studies were only carried out for highly familiar grasps whose successful completion would require little if any visual feedback.

5 258 E. Oztop et al. / Neural Networks 19 (2006) in RNNPB; but rather (core) primitive dynamical systems are adapted to match the demonstrated movements using local learning techniques (Ijspeert et al., 2003; Schaal et al., 2004). In this article, we focus on RNNPB as the representative of dynamical system approaches since the authors have already hinted that RNNPB captures some properties of the mirror neurons (Tani et al., 2004). The RNNPB has three operational modes; we review each of them starting from the learning mode Learning mode The learning is performed in an off-line fashion by providing the sensory-motor training stimuli (e.g. two trajectories one for the position of a moving hand and the other for the joint angles of the arm) for each behavior in the training set. The goal of the training is twofold (see Fig. 1): (1) to adapt weight sets w and W such that the network becomes a time series predictor for the sensory-motor stimuli, and (2) to create PB vectors for each training behavior. Both adaptations are based on the prediction error; the weights w and W are adapted by back-propagation over all the training patterns as in a usual recurrent neural network. The PB vectors, however, are updated separately for each training pattern for reducing the prediction error. Furthermore, the modulation of PB vectors are kept slow to obtain a fixed PB vector for each learnt behavior Action generation mode After learning, the model represents a set of behaviors as dynamical systems tagged by the PB vectors created during the learning phase. The behaviors are generated via the associated PB vectors. Given a fixed PB vector, the network autonomously produces a sensory motor stream corresponding to the behavior associated with the PB vector (see Fig. 2). Note that the behavior generation mode of RNNPB requires the sensory motor prediction to be fed back into the sensory motor input as shown in Fig. 2. Fig. 2. The RNNPB in behavior generation mode. Given a fixed PB vector (thick arrow) the network produces the corresponding stored sensory motor stream Action recognition mode The task of the network in this mode is to observe an ongoing behavior (sensory data) and compute a PB vector that is associated with a behavior that matches the observed one as much as possible. The arrival of a sensory input generates a prediction of the next sensory stimulus at the output layer. Then the actual next sensory input is compared with this prediction creating a prediction error (see Fig. 3). The prediction error is back-propagated to the PB vectors (i.e. PB vectors are updated such that the prediction error is reduced). The actual computation of the PB vectors is performed using the so-called regression window of the past steps so that the change of PB vectors can be constrained to be smooth over the window. The readers are referred to Tani et al. (2004) for further details of this step. If sensory input matches one of the learned behaviors, the PB vectors tend to converge to the values determined during learning (Tani et al., 2004). Note that in this mode, the feedback from sensory motor output to sensory-motor input is restricted only to the motor component Relation to MNs and imitation The model has been shown to allow a humanoid robot to imitate and learn actions via demonstration. The logical link Fig. 1. The RNNPB network in learning phase. The weights W and w are adapted so as to reduce the prediction error over all the behaviors to be learnt (temporal sensory motor patterns). The PB vectors are also updated to reduce the prediction error, albeit separately for each behavior and at a slower rate to ensure representative PB values for each behavior. Fig. 3. The RNNPB in behavior recognition mode. In the recognition mode the sensory input is obtained from external observation (thick arrows). The feedback from sensory motor output to sensory motor input is restricted only to the motor component. The prediction error is used to compute the parametric bias vector corresponding to the incoming sensory data.

6 E. Oztop et al. / Neural Networks 19 (2006) to mirror neurons comes from the fact that the system works as both a behavior recognizer and generator after learning. PB units are tightly linked with the behavior being executed or observed. During execution, a fixed PB vector selects one of the stored motor patterns. For recognition PB unit outputs iteratively converge to the action observed. Although mirror neurons do not determine the action to be executed in monkeys (Fogassi et al., 2001), the firing patterns of mirror neurons are correlated with the action being executed. Thus, PB vector units may be considered analogous to mirror neurons. Ito and Tani (2004) suggest that the PB units activities should be under control of a higher mechanism to avoid unwanted imitation, such as for dangerous movements. One prediction, or rather a question posed to neurophysiology, is what happens when a dangerous movement is observed by a monkey. Although it is known that (initially) F5 mirror neurons do not respond to unfamiliar actions, no data on the parietal mirror neurons exists to rule out this possibility. However, we again emphasize that mirror response does not automatically involve movement imitation and so it is unlikely that the monkey mirror neuron system is an inhibited imitation system ; rather imitation ability must have been developed on top of the mirror neuron system along the course of primate evolution. 7. Motor learning and imitation: a modular architecture The RNNBP of the previous section represents multiple behaviors as a distributed code. Demiris and Hayes (2002) and Demiris and Johnson (2003) chose the opposite approach representing each behavior as a separate module in their proposed imitation system. Following the organization of the MOSAIC model (Wolpert & Kawato, 1998) (see Section 12), the key structure of the proposed architecture is formed by a battery of behaviors (modules) paired with forward models, where each behavior module receives information about the current state (and possibly the target goal), and outputs the motor commands that is necessary to achieve the associated behavior (see Fig. 4). A forward model receives output of the paired behavior module and estimates the next state which is fed back to the behavior module Fig. 4. The imitation architecture proposed by Demiris and Hayes (2002) and Demiris and Johnson (2003) is composed of a set of paired behavior and forward models. During imitation mode, the comparison of the predicted next state (of the demonstrator) with the actual observed state gives an indication of which behavior module should be active for the correct imitation of the observed action. for parameter adjustments. A behavior is similar to an inverse model, although inverse models do not usually utilize feedback, but output commands in a feed-forward manner. However, the boundary between behavior and inverse model is not a rigid one (Demiris & Hayes, 2002). The architecture implements imitation by assuming that the demonstrator s current state (e.g. joint angles of a robot) is available to it. When the demonstrator executes a behavior, the perceived states are fed into the imitator s available behavior modules in parallel which generate motor commands that are sent to the forward models. The forward models predict the next state based on the incoming motor commands, which are then compared with the actual demonstrator s state at the next time step. The error signal resulting from this comparison is used to derive a confidence value for each behavior (module). The behavior with the highest confidence value (i.e. the one that best matches the demonstrator s behavior) is selected for imitation. When an observed behavior is not in the existing repertoire, none of the existing behaviors reach a high confidence value, thus indicating that a new behavior should be added to the existing behavior set. This is achieved by extracting representative postures while the unknown behavior is demonstrated, and constructing a behavior module (e.g. a PID controller) to go through the representative postures extracted. This computational procedure to estimate the other agent s behavioral module by simulating one s own forward model and controller is essentially identical to the proposal by Doya, Katagiri, Wolpert, and Kawato (2000). Demiris and Simmons (this issue) describe a hierarchical architecture that employs similar principles at its core. Although at a conceptual level, this architecture has strong parallels with the MOSAIC model (Haruno et al., 2001; Wolpert & Kawato, 1998), MOSAIC takes learning and control as the core focus by providing explicit learning mechanisms (see Section 12) Relation to MNs and imitation The architecture can be related to mirror neurons because the behavior modules are active during both movement generation and observation. However, all the modules are run in parallel in the proposed architecture, so it is more reasonable to take the confidence values as the mirror neuron responses. Demiris and Hayes (2002) and Demiris and Johnson (2003) arrived at several predictions about mirror neurons albeit considering imitation-ability and mirror neuron activity interchangeable, while we think they must be analyzed separately. One interesting prediction which has also been predicted by the MNS model (Oztop & Arbib, 2002) (see Section 10) is the following. A mirror neuron which is active during the demonstration of an action should not be active (or possibly be less active) if the demonstration is done at speeds unattainable by the monkey. A further prediction states that mirror neurons that remain active for a period of time after the end of the demonstration are encoding more complex sequences that incorporate the demonstration as their first part (Demiris & Hayes, 2002). The other predictions implied by the structure of the architecture are the existence of other

7 260 E. Oztop et al. / Neural Networks 19 (2006) goal directed mirror neurons and the trainability of new mirror neurons. 8. An evolutionary approach The evolutionary algorithms that Borenstein and Ruppin (2005) used to explain mirror neurons and imitation are quite different from the previous approaches. Evolutionary algorithms incorporate aspects of natural selection (survival of the fittest) to solve an optimization problem. An evolutionary algorithm maintains a population of structures ( individuals ) that evolves according to rules of selection, recombination, mutation and survival. A shared world determines the fitness or performance of each individual and identifies the optimization problem. Each generation is composed of fitter individuals and their variants, while fewer not-fit individuals and their variants are allowed to reproduce their traits. After many generations one expects to find a set of high performing individuals which represent close-tooptimal solutions to the original problem. Within this framework, Borenstein and Ruppin (2005) defined individuals as simple neuro-controllers that could sense the state of the world and the action of a teaching agent (inputs) and generate actions (outputs). Individuals generated output with a simple 1-hidden-layer feedforward neural network. The fitness of an individual was defined as a random mapping from world state to actions, which was kept fixed for the lifetime of an individual. The evolutionary encoding (genes) determined the properties of individual network connections (synapses): type of learning, initial strength, either inhibitory or excitatory type, and rate of plasticity Relation to MNs and imitation The simulation with populations of 200 individuals evolving for 2000 generations showed interesting results. The best individuals developed neural controllers that could learn to imitate the teacher. Furthermore, the analysis of the units in the hidden layer of these neuro-controllers revealed units which were active both when observing the teacher and when executing the correct action, although not all the actions were mirrored. The conclusion drawn was that there is an essential link between the ability to imitate and a mirror system. Despite the interest of the demonstration that evolving neural circuitry to imitate yields something like mirror neurons as a byproduct, we note again that even though monkeys have mirror neurons they are not natural imitators. Thus, the evolution of mirror systems on the basis of evolutionary pressure to imitate does not seem to capture the time course of primate evolution. 9. Associative memory hypothesis of mirror neurons In this section, we avoid choosing a single architecture for extensive review since the core mechanism employed in all the candidate models relies on a very simple principle deriving from the classical view of Hebbian synaptic plasticity in the cerebral cortex. Implementation of this view results in connectionist architectures referred as associative or content addressable Fig. 5. A generic associative memory for an agent (a robot or an organism). When an agent generates a movement using motor code, the sensed stimuli are associated with this code. At a later time a partial representation of associated stimuli (e.g. vision) can be used to retrieve the whole (including the motor code). The connectivity among the units representing different modalities could be full or sparse. memories (Hassoun, 1993), to which the models reviewed in this section more or less conform. The crucial feature of an associative memory is that a partial representation of a stored pattern is sufficient to reconstruct the whole. In general a neural network which does not distinguish input and output channels can be considered as an associative memory (with possibly hidden units). Fig. 5 schematizes a possible association that can be established when a biological or an artificial agent acts. The association can take place among the motor code, the somatosensory, vestibular, auditory and visual stimuli sensed when the movement takes place with the execution of the motor code. If we hypothesize that mirror neurons are part of a similar mechanism then the mirror neuron responses could be explained: when the organism generates motor commands the representation of this command and the sensed (somatosensory, visual and auditory) effects of the command are associated within the mirror neuron system. Then at a later time when the system is presented with a stimulus that partially matches one of the stored patterns (i.e. vision or audition of an action alone) the associated motor command representation is retrieved automatically. This representation can be used (with additional circuitry) to mimic the observed movement. This line of thought has been explored through robotic implementations of imitation using a range of associative memory architectures. Elshaw, Weber, Zochios, and Wermter (2004) implemented an associator network based on the Helmholtz machine (Dayan, Hinton, Neal, & Zemel, 1995) where the motor action codes were associated with vision and language representations. The learned association enabled neurons of the hidden layer of the network to behave as mirror neurons; the hidden units could become active with one of motor, vision or language inputs. Kuniyoshi, Yorozu, Inaba, and Inoue (2003) used a spatio-temporal associative memory called the non-monotone neural net (Morita, 1996) to associate self generated arm movements of a robot with the local visual flow generated. Billard and Mataric (2001) used the DRAMA architecture (Billard & Hayes, 1999), which is a time-delay recurrent neural network with Hebbian

8 E. Oztop et al. / Neural Networks 19 (2006) update dynamics at the core of their biologically inspired imitation architecture. Oztop, Chaminade, Cheng, and Kawato (2005) used an extension of a Hopfield net utilizing product terms to implement a hand posture imitation system using a robotic hand Relation to MNs and imitation In spite of the differences in the implementation, the common property among the aforementioned associative memory models is the multi-modal activation of the associative memory/network units. Thus, when these models are considered as models of mirror neurons (note that not all models claim to be models for mirror neurons, as the main focus is on imitation) then the explanation of the existence of mirror neurons becomes phenomenological rather than functional (see Section 13). For example, the models of Kuniyoshi et al. (2003) and Oztop et al. (2005) use selfobservation as the principle for bootstrapping imitation and formation of units that respond to self-actions and observations of others. We refer to this type approach as ASSOC so that we can collectively refer to these models in Section 13, where we propose a taxonomy based on the modeling methodology. 10. Mirror neuron system (MNS) model: a developmental view The models reviewed so far related to the mirror neurons indirectly, through imitation. Here, we present a computational model with anatomically justified connectivity, which directly explores how MNs develop during infancy. It is quite unlikely that the MNs are innate because mirror neurons have been observed for tearing paper, for instance (Kohler et al., 2002). With this observation, the MNS model (Oztop & Arbib, 2002) takes a developmental point of view and explains how the mirror neurons are developed during infancy. The main hypothesis of the model is that the temporal profile of the features an infant experiences during self-executed grasps provides the training stimuli for the mirror neuron system to develop. 6 Thus, developmentally, grasp learning precedes initial mirror neuron formation. 7 Although MNS proposes that MNs are initially evolved to support motor control, it does not provide computational mechanisms showing this The Model MNS is a systems level model of the (monkey) mirror neuron system for grasping. The computational focus of the model is the 6 Only grasp related visual mirror neurons were addressed. A subsequent study (Bonaiuto et al., 2005) has introduced a recurrent network learning architecture that not only reproduces key results of Oztop and Arbib (2002) but also addresses the data of Umiltá et al. (2001) on grasping of recently obscured objects and of Kohler et al. (2002) on audiovisual mirror neurons. 7 Note again that monkeys have a mirror system but do not imitate. It is thus a separate question to ask How, in primates that do imitate, does the imitation system build (both structurally and temporally) on the mirror system?. development of mirror neurons by self-observation; the motor production component of the system is assumed to be in place and not modeled using neural modules. The schemas 8 (Arbib, 1981) of the model are implemented with different level of granularity. Conceptually those schemas correspond to brain regions as follows (see Fig. 6). The inferior premotor cortex plays a crucial role when the monkey itself reaches for an object. Within the inferior premotor cortex area F4 is located more caudally than area F5, and appears to be primarily involved in the control of proximal movements (Gentilucci et al., 1988), whereas the neurons of F5 are involved in distal control (Rizzolatti et al., 1988). Areas IT (inferotemporal cortex) and cips (caudal intraparietal sulcus) provide visual input concerning the nature of the observed object and the position and orientation of the object s surfaces, respectively, to AIP. The job of AIP is to extract the affordances the object offers for grasping. By affordance we mean the object properties that are relevant for grasping such as the width, height and orientation. The upper diagonal in Fig. 6 corresponds to the basic pathway AIP/F5 canonical/m1 (primary motor cortex) for distal (reach) control. The lower right diagonal (MIP/LIP/VIP/F4) of Fig. 6 provides the proximal (reach) control portion of the MNS model. The remaining modules of Fig. 6 constitute the sensory processing (STS and area 7a) and the core mirror circuit (F5 mirror and area 7b). Mirror neurons do not fire when the monkey sees the hand movement or the object in isolation; the sight of the hand moving appropriately to grasp or manipulate a seen (or recently seen) object is necessary for the mirror neurons tuned to the given action to fire (Umiltá et al., 2001). This requires schemas for the recognition of the shape of the hand and the analysis of its motion (performed by STS in the model), and for the analysis of the hand-object relation (area 7a in Fig. 6). The information gathered at STS and areas 7a are captured in the hand state at any instant during movement observation and serves as an input to the core mirror circuit (F5 mirror and area 7b). Although visual feedback control was not built into MNS, the hand state components track the position of hand and fingers relative to the object s affordance (see Oztop and Arbib (2002) for the full definition of the hand state) and can thus be used in monitoring the successful progress of a grasping action supporting motor control. The crucial point is that the information provided by the hand state allows action recognition because relations encoded in the hand state form an invariant of the action regardless of the agent of the action. This allows self-observation to train a system that can be used for detecting the actions of others and recognizing them as one of the actions of the self. During training, the motor code represented by active F5 canonical neurons was used as the training signal for the core mirror circuit to enable mirror neurons to learn which hand-object trajectories corresponded to the canonically 8 A schema refers to a functional unit that can be instantiated as a modular unit, or as a mode of operation of a network of modules, to fulfill a desired input/output requirement (Arbib, 1981; Arbib et al., 1998).

9 262 E. Oztop et al. / Neural Networks 19 (2006) Fig. 6. A schematic view of the mirror neuron system (MNS) model. The MNS model learning mechanisms and simulations focus on the core mirror circuit marked by the central diagonal rectangle (7b and F5 mirror), see text for details (Oztop & Arbib, 2002; reproduced with kind permission of Springer Science and Business Media). encoded grasps. We reiterate that the input to the F5 mirror neurons is not the visual stimuli as created by the hand and the object in the visual field but the hand state trajectory (trajectory of the relation of the hand and the object) extracted from these stimuli. Thus, training tunes the F5 mirror neurons to respond to hand-object relational trajectories independent of the owner of the action ( self or other ) Relation to MNs The focus of the simulations was the 7b-F5 complex (core mirror circuit). The input and outputs of this circuit was computed using various schemas providing a context to analyze the circuit. The core mirror circuit was implemented as a feedforward neural network (1-hidden layer backpropagation network with sigmoidal activation units; hidden layer: area 7b; output layer: F5 mirror) responding to increasingly long initial segments of the hand-state trajectory. The network could be trained to recognize the grasp type from the hand state trajectory, with correct classification often being achieved well before the hand reached the object. For the preprocessing and training details the reader is referred to Oztop and Arbib (2002). Despite the use of a non-physiological neural network, simulations with the model generated a range of predictions about mirror neurons that suggest new neurophysiological experiments. Notice that the trained network responded not only to hand state trajectories from the training set, but also showed interesting responses to novel grasping modes. For example Fig. 7 shows one prediction of the MNS model. An ambiguous precision pinch grasp activates multiple neurons (power and precision grasp responsive neurons) during the early portion of the movement observation. Only later does the activity of the precision pinch neuron dominate and the power grasp neuron s activity diminish. Other predictions were derived from the spatial perturbation experiment where the hand did not reach the goal (i.e. a fake grasp), and the altered kinematics experiment where the hand moved with constant velocity profile. The former case showed a non-sharp decrease in the mirror neuron activity while the latter showed a sharp decrease. The reader is referred to Oztop and Arbib (2002) for the details and other simulation experiments. Recently, Bonaiuto, Rosta, and Arbib (2005) developed the MNS2 model, a new version of the MNS model of action recognition learning by mirror neurons of the macaque brain, using a recurrent architecture that is biologically more plausible than that of the original model. Moreover, MNS2 extends the capacity of the model to address data on audio visual mirror neurons (Kohler et al., 2002) and on response of mirror neurons when the target object was recently visible but is currently hidden (Umiltá et al., 2001). 11. The mental state inference (MSI) model: forward model hypothesis for MNs The anatomical location (i.e. premotor cortex) and motor response of mirror neurons during grasping suggest that the fundamental function of mirror neurons may be rooted in grasp control. The higher cognitive functions of mirror neurons, then should be seen as a later utilization of this system, augmented with additional neural circuits. Although MNS of the previous section adopted this view, it did not model the motor component, which is addressed by the MSI model.

10 E. Oztop et al. / Neural Networks 19 (2006) Fig. 7. Power and precision grasp resolution. (a) The left panel shows the initial configuration of the hand while the right panel shows the final configuration of the hand, with circles showing positions of the wrist in consecutive frames of the trajectory. (b) The distinctive feature of this trajectory is that the hand initially opens wide to accommodate the length of the object, but then thumb and forefinger move into position for a precision grip. Even though the model had been trained only on precision grips and power grips separately, its response to this input reflects the ambiguities of this novel trajectory the curves for power and precision cross towards the end of the action, showing the resolution of the initial ambiguity by the network. (Oztop & Arbib, 2002, reproduced with kind permission of Springer Science and Business Media) Visual feedback control of grasping and the forward model hypothesis for the mirror neurons The mental state inference (MSI) model builds upon a visual feedback circuit involving the parietal and motor cortices, with a predictive role assigned to mirror neurons in area F5. For understanding others intentions, this circuit is extended into a mental state inference mechanism (Oztop et al., 2005). The global functioning of the model for visual feedback control proceeds as follows. The parietal cortex extracts visual features relevant to the control of a particular goal-directed action (X, the control variable) and relays this information to the premotor cortex. The premotor cortex computes the motor signals to match the parietal cortex output (X) to the desired neural code (X des ) relayed by prefrontal cortex. The desired change generated by the premotor cortex is relayed to dynamics related motor centers for execution (Fig. 8, upper panel). The F5 mirror neurons implement a forward prediction circuit (forward model) estimating the sensory consequences of F5 motor output related to manipulation, thus compensating for the sensory delays involved in the visual feedback circuit. This is in contrast to the generally suggested idea that mirror neurons serve solely to retrieve an action representation that matches the observed movement. During observation mode, these F5 mirror neurons are used to create motor imagery or mental simulation of the movement for mental state inference (see below). Although MSI does not specify the region within parietal cortex that performs control variable computation, recent findings suggest that a more precise delineation is possible. Experiments with macaque monkeys indicate that parietal area PF (area 7b) may be involved in monitoring the relation of the hand with respect to an object during grasping. Some of the PF neurons that do not respond to vision of objects become active when the monkey (without any arm movement) watches movies of moving hands (of the experimenter or the monkey) for manipulation, suggesting that the neural responses may reflect the visual feedback during observed hand movements (Murata, 2005). It is also possible that a part of AIP may be involved in monitoring grasping as shown by transcranial magnetic stimulation (TMS) with humans (Tunik, Frey, & Grafton, 2005). As in the MNS model, area F5 (canonical) is involved in converting the parietal output (PF/AIP) into motor signals, which are used by primary motor cortex and spinal cord for actual muscle activation. In other words, area F5 nonmirror neurons implement a control policy (assumed to be learned earlier) to reduce the error represented by area PF/AIP output Mental state inference The ability to predict enables the feedback circuit of Fig. 8 (upper panel) to be extended into a system for inferring the intentions of others based on the kinematics of goal directed actions (see Fig. 8 lower panel). In fact, the full MSI model involves a mental simulation loop that is built around a forward model (Blakemore & Decety, 2001; Wolpert & Kawato, 1998), which in turn is used by a mental state inference loop to estimate the intentions of others. The MSI model is described for generic goal-directed actions, however here we look at the model in relation to a tool grasping framework where two agents can each grasp a virtual hammer with different intentions (holding, nailing or prying a nail). Depending on the planned subsequent use of a hammer, grasping requires differential alignment of the hand and the thumb. Thus, the kinematics of the action provides information about the intention of the actor. For this task, the mental state was modeled as the intention in grasping the hammer. Within this framework an observer guesses the target object (one of various objects in the demonstrator s workspace) and the type of grasp and produce an appropriate F5 motor signal that is inhibited for actual muscle activation but used by the forward model (MNs) (see Fig. 8 lower panel). With the sensory outcome predicted by the MNs, the movement can be simulated as if it were executed in an online feedback mode. The match of the simulated sensations of a simulated movement with the sensation of observed movement will then signal the correctness of the guess. The simulated mental sensations and actual perception of movement is compared in a