A quick overview of LOCEN-ISTC-CNR theoretical analyses and system-level computational models of brain: from motivations to actions

Transcription

1 A quick overview of LOCEN-ISTC-CNR theoretical analyses and system-level computational models of brain: from motivations to actions Gianluca Baldassarre Laboratory of Computational Neuroscience, Institute of Cognitive Sciences and Technologies, Italian National Research Council (ISTC-CNR) 1. Introduction: the purposes of this brief document The goal of this brief document is to give a focussed overview of the theoretical neuroscience and computational modelling work carried out by the Research Group Laboratory of Computational Embodied Neuroscience (LOCEN) of the ISTC-CNR ( The overview focusses on the work of LOCEN directed to understand the system-level overall organisation of brain that allows organisms, in particular primates, to express multiple motor behaviours sub-serving multiple needs, and how such behaviors are acquired through learning processes. This overall goal of LOCEN involves some important topics of current cognitive science, in particular: (a) the goaldirected/habitual hierarchical organisation of behaviour; (b)

2 the organisation of motivational systems at the basis of the learning and selection of motor behaviours. 2. Computational models of goal-directed and habitual behaviour Instrumental behaviour is a fundamental means through which animals, and in particular primates, flexibly and adaptively accomplish their multiple needs with variable internal and external conditions. Instrumental behaviour can be subdivided in habitual and goal-directed behavior. Habitual behaviour brain mechanisms link specific stimuli to instrumetal behaviours (S-R), i.e. behaviours that are not innate but have been acquired by trial-and-error processes (Mannella and Baldassarre 2007; Ciancio et al. 2013; Caligiore et al. 2014). Goal-directed behaviour mechanisms, instead, link desired outcomes to instrumental actions (A-O) (Baldassarre et al. 2013), for which the triggering of behaviours is dependent on the animal's relative desirability of those outcomes given its current needs and stimuli in the world (Mannella et al. 2010). The acquisition and expression of multiple behaviours relies on two highy integrated brain systems (Baldassarre and Mirolli 2013; Baldassarre et al. 2013; Thill et al. 2013). The first is related to the two main cortico-cortical pathways of brain (Caligiore et al. 2010): (a) the dorsal pathway that, from visual areas and passing through parietal and premotor cortex, reaches motor areas: this system subserves the encoding of affordances and the on-line guidance of action (Caligiore et al. 2010); (b) the ventral pathway, that from visual areas, reaches the temporal cortex, encoding the identity of objects/resources, and then the prefrontal cortex: this system encods the animal's goals (Thill et al. 2013) and implements more sophisticated processes such as planning (Baldassarre 2003; Pezzulo et al. 2007) and imaging (Seepanomwan et al. 2013). The second brain system is formed by the basal gangliacortical loops, anatomically `intercepting' the late stages of the two cortical pathways, and formed by (Baldassarre et al. 2013; Chersi et al. 2013): (a) the ventral basal gangliaprefrontal cortex loop, supporting outcome/goal selection; (b)

3 the medial basal ganglia-associative cortex loop, subserving the selection of the contents of associative cortex e.g., attentional targets, affordances, and objects identity, and goal-action contingencies; (c) the dorsal basal ganglia-motor cortex loop, supporting the trial-and-error learning and selection of actions (Baldassarre 2002; Caligiore et al. 2014) possibly in concert with the cerebellum (Caligiore et al. 2013). 3. Motivational systems behind motivation and learning Ventral striatum is a key nexus between the generation of motivational value of action outcomes (e.g., ingested foods), and the selection of goals (e.g., seen foods) implemented by the ventral basal ganglia-prefrontal cortex loops (Mannella et al., 2013). The values of outcomes can be generated by extrinsic motivations, related to the attainment of resources increasing the animal's biological fitness and involving areas such as amygdala (Mannella et al. 2010; Mirolli et al. 2013). This generation of value is based on Pavlovian processes (underlying the triggering of innate reactions towards the brain, body, and world, Mirolli et al. 2013) and their close interplay with instrumental behaviour (Cartoni et al. 2013). Alternatively, values can be generated by intrinsic motivations (Baldassarre and Mirolli, 2013; Baldassarre et al. 2014), related to knowledge and skills acquisition (Baldassarre 2011) and linked to the hippocampus, detecting the novelty or surprise of stimuli (Barto et al. 2013), and basal ganglia-prefrontal cortex processes, related to competence aquisition and agency (Schembri et al. 2007; Santucci et al. 2013; Polizzi et al. 2014; Taffoni et al. 2014). The assignment of value to possible action outcomes and goals strongly relies on the production of neuromodulators, such as dopamine and noradrenaline, regulating the overall brain functioning (Fiore et al., 2013), the basal ganglia selection processes (Fiore et al. 2013a), and intrinsic and extrinsic learning processes (Mirolli et al. 2013).

4 4. Conclusions The overall organisation of brain architecture and processes subserving primates' flexible behaviour is very complex. However, the work of LOCEN-ISTC-CNR reviewed here shows that its interdisciplinary investigation, pivoting on system-level computational models and computationally informed theoretical analyses of empirical evidence, can lead to identify relatively few main architectural and functioning principles underlying them. Bibliografia Baldassarre, G. (2002), A modular neural-network model of the basal ganglia's role in learning and selecting motor behaviours, Journal of Cognitive Systems Research, 3(2), Baldassarre, G. (2003). Forward and bidirectional planning based on reinforcement learning and neural networks in a simulated robot. In Butz, M., Sigaud, O., Gérard, P. (eds.), Anticipatory behaviour in adaptive learning systems, pp Berlin: Springer. Baldassarre, G. (2011). What Are Intrinsic Motivations? A Biological Perspective. In ICDL-EpiRob E1-8. New York, NY: IEEE. Baldassarre, G., Caligiore, D., Mannella, F. (2013). The hierarchical organisation of cortical and basal-ganglia systems: a computationally-informed review and integrated hypothesis. In Baldassarre, G., Mirolli, M. (eds.), Computational and Robotic Models of the Hierarchical Organisation of Behaviour Berlin: Springer-Verlag. Baldassarre, G., Mirolli, M. (2013)(eds.). Computational and Robotic Models of the Hierarchical Organisation of Behaviour. In Baldassarre, G., Mirolli, M. (ed.), Berlin: Springer-Verlag. Baldassarre, G., Mirolli, M. (2013)(eds.). Intrinsically motivated learning in natural and artificial systems. In Baldassarre, G., Mirolli, M. (eds.), Berlin: Springer. Baldassarre, G., Mannella, F., Fiore, V. G., Redgrave, P., Gurney, K., Mirolli, M. (2013), Intrinsically motivated action-outcome learning and goal-based action recall: A system-level bio-constrained computational model. Neural Networks, 41, Baldassarre, G., Stafford, T., Mirolli, M., Redgrave, P., Ryan, R., Barto, A. (2014). Overview of the Frontiers Research Topic

5 `Intrinsic motivations and open-ended development in animals, humans, and robots'. Frontiers in Psychology, 5 (985), e1-5. Frontiers. Barto, A., Mirolli, M., Baldassarre, G. (2013), Novelty or surprise?, Frontiers in Psychology Cognitive Science, 4(907), Caligiore, D., Borghi, A., Parisi, D., Baldassarre, G. (2010), TRoPICALS: A Computational Embodied Neuroscience Model of Compatibility Effects, Psychological Review, 117(4), Caligiore, D., Borghi, A., Parisi, D., Ellis, R., Cangelosi, A., Baldassarre, G. (2013), How affordances associated with a distractor object affect compatibility effects: A study with the computational model TRoPICALS, Psychological Research, 77(1), Caligiore, D., Parisi, D., Baldassarre, G. (2014), Integrating Reinforcement Learning, Equilibrium Points and Minimum Variance to Understand the Development of Reaching: A Computational Model, Psychological Review, 121(3), Caligiore, D., Pezzulo, G., Miall, C., Baldassarre, G. (2013), The contribution of brain sub-cortical loops in the expression and acquisition of action understanding abilities, Neuroscience and Biobehavioral Reviews, 37(10), Cartoni, E., Puglisi-Allegra, S., Baldassarre, G. (2013), The three principles of action: a Pavlovian-instrumental transfer hypothesis, Frontiers in Behavioural Neuroscience 7(153), e1 11. Caligiore, D., Tommasino, P., Sperati, V., Baldassarre, G. (2014). Modular and hierarchical brain organization to understand assimilation, accommodation and their relation to autism in reaching tasks: a developmental robotics hypothesis. Adaptive Behavior. Chersi, F., Mirolli, M., Pezzulo, G., Baldassarre, G. (2013), A spiking neuron model of the cortico-basal ganglia circuits for goaldirected and habitual action learning, Neural Networks, 41, Ciancio, A. L., Zollo, L., Baldassarre, G., Caligiore, D., Guglielmelli, E. (2013), The Role of Learning and Kinematic Features in Dexterous Manipulation: a Comparative Study with Two Robotic Hands, International Journal of Advanced Robotic Systems, 10, e1-21. Fiore, V. G., Mannella, F., Mirolli, M., Latagliata, E. C., Valzania, A., Cabib, S., Dolan, R. J., Puglisi-Allegra, S., Baldassarre, G. (2014), Corticolimbic catecholamines in stress: a computational model of the appraisal of controllability, Brain Structure and Function, e1-15.

6 Fiore, V. G., Sperati, V., Mannella, F., Mirolli, M., Gurney, K., Firston, K., Dolan, R. J., Baldassarre, G. (2014a), Keep focussing: striatal dopamine multiple functions resolved in a single mechanism tested in a simulated humanoid robot, Frontiers in Psychology Cognitive Science, 5 (124), e1-17. Mannella, F., Baldassarre, G. (2007), A neural-network reinforcement-learning model of domestic chicks that learn to localize the centre of closed arenas, Philosophical Transactions of the Royal Society B Biological Sciences, 362(1479), Mannella, F., Mirolli, M., Baldassarre, G. (2010). The interplay of Pavlovian and instrumental processes in devaluation experiments: a computational embodied neuroscience model tested with a simulated rat. In Tosh, C., Ruxton, G. (eds.), Modelling Perception With Artificial Neural Networks, pp Cambridge: Cambridge University Press. Mannella, F., Gurney, K., Baldassarre, G. (2013), The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis., Front Behav Neurosci, 7(135), e1-29. Mirolli, M., Mannella, F., Baldassarre, G. (2010). The roles of the amygdala in the affective regulation of body, brain and behaviour. Connection Science, 22 (3), Mirolli, M., Baldassarre, G., Santucci, V. G. (2013), Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcement driving both action acquisition and reward maximization: A simulated robotic study, Neural Networks, 39, Pezzulo, G., Baldassarre, G., Butz, M. V., Cristiano, C., Hoffmann, J. (2007). From actions to goals and vice-versa: theoretical analysis and models of the ideomotor principle and TOTE. In Butz, M. V., Sigaud, O., Pezzulo, G., Baldassarre, G. (eds.), Anticipatory Behavior in Adaptive Learning Systems: From Brains to Individual and Social Behavior, pp Berlin: Springer. Polizzi di Sorrentino, E., Sabbatini, G., Truppa, V., Bordonali, A., Taffoni, F., Formica, D., Baldassarre, G., Mirolli, M., Guglielmelli, Eugenio, V. E. (2014), Exploration and learning in capuchin monkeys (Sapajus spp.): the role of actionoutcome contingencies, Animal Cognition, 17(5), Santucci, V. G., Baldassarre, G., Mirolli, M. (2013), Which is the best intrinsic motivation signal for learning multiple skills?, Frontiers in Neurorobotics, 7, e1-14. Schembri, M., Mirolli, M., Baldassarre, G. (2007). Evolving childhood s length and learning parameters in an intrinsically

7 motivated reinforcement learning robot. In EpiRob 2007, pp Seepanomwan, K., Caligiore, D., Baldassarre, G., Cangelosi, A. (2013), Modelling mental rotation in cognitive robots, Adaptive Behavior, 21(4), Taffoni, F., Tamilia, E., Focaroli, V., Formica, D., Ricci, L., Di Pino, G., Baldassarre, G., Mirolli, M., Guglielmelli, E., Keller, F. (2014), Development of goal-directed action selection guided by intrinsic motivations: an experiment with children, Experimental Brain Research, 232(7), Thill, S., Caligiore, D., Borghi, A. M., Ziemke, T., Baldassarre, G. (2013), Theories and computational models of affordance and mirror systems: An integrative review, Neuroscience and Biobehavioral Reviews, 37,