The computational and neural basis of voluntary motor control and planning

Transcription

1 Review The computational and neural basis of voluntary motor control and planning Stephen H. Scott Centre for Neuroscience Studies, Department of Biomedical and Molecular Sciences, Department of Medicine, Queen s University, Kingston, ON K7L 3N6, Canada Optimal feedback control (OFC) provides a powerful tool to interpret voluntary motor control, highlighting the importance of sensory feedback in the control and planning of movement. Recent studies in the context of OFC have increasingly used mechanical perturbations and visual shifts to probe voluntary control processes. These studies reveal the surprising sophistication of corrective responses, which are goal-directed and exhibit knowledge of the physical properties of the limb and the environment. These complex feedback processes appear to be generated through transcortical feedback pathways. The research reviewed here opens and enhances several lines of discovery, including testing whether feedback corrections share all of the attributes associated with voluntary control, identifying how prediction influences optimal state estimation, and importantly, how these voluntary control processes are generated by the highly distributed circuitry within the brain. Changing views on the use of sensory feedback for voluntary control It is easy to recognize that sensory signals, such as from our eyes, skin or muscles, help us perceive our body and the world around us. It is harder to understand how these same sensory signals help guide and correct our body movements. Sensory feedback is clearly involved when someone accidentally bumps your arm at a cocktail party leading to rapid motor corrections to avoid spilling a drink held in the hand. Less obvious is whether and how sensory feedback is used continuously to adjust small errors when we reach out to grab an object. The importance of sensory feedback for motor control was demonstrated over a century ago in the classic study by Sherrington on the influence of afferent feedback on spinal level reflexes in the cat hindlimb [1]. The development of feedback control theory in the mid 2 th century led to its application to biological control [2] and studies in the 197s began to explore spinal and cortical feedback during voluntary motor actions. However, several observations in the 198s suggested that feedback may not be very important for voluntary control [3]. In particular, it was noted that transmission delays between the central nervous system and the limb, on the order of 1s of milliseconds, makes traditional servo-control unstable. As a result, voluntary control was commonly assumed to use motor programs, with motor cortex specifying feedforward Corresponding author: Scott, S.H. (steve.scott@queensu.ca). commands to the spinal cord. Feedback was relegated to a modest contribution at only the spinal level. Recognition of the importance of sensory feedback for voluntary control is in a renaissance, due to the introduction of optimal feedback control (OFC) as a model of biological control. Here I review the use of OFC to interpret voluntary control and the value of using perturbations (once again) to probe the properties of the voluntary motor system. Finally, I provide a brief sketch of how OFC-like control may be implemented by distributed circuits in the brain. Optimal Feedback Control (OFC) as a theory of voluntary control Optimality principles have been common in motor control with various possible objectives, such as minimizing jerk, torque change, or the influence of noise [4]. In 22, Todorov and Jordan [5] proposed OFC as a theory of voluntary control (Figure 1a). Delayed sensory feedback that makes servo-control unstable is overcome by using optimal state estimation, a Kalman filter integrating efference copy signals with delayed sensory feedback. The derivation of the optimal control policy (i.e., feedback gains) uses knowledge of the system dynamics, such as properties of the musculoskeletal system, to achieve an optimal balance between behavioural performance and associated motor costs [6]. What is most impressive about these controllers is that they capture many aspects of biological control. In particular, biological movements are highly variable and yet always quite successful. This variability is created by noise (neural, mechanical, and other), a known feature of sensorimotor systems [7]. Importantly, these types of controllers correct errors/noise if they influence the goal of the task and ignore them if they do not interfere with the goal. This selective control of movement leads to characteristic patterns of motor variability during behavior [8 13]. Although other models of motor control exist [14 16], they cannot capture this behavioural-level feature of voluntary control. One criticism of OFC as a model for biological control is that it cannot be disproved; observed behaviour can be predicted by modifying the cost function (this is true to a degree). There are studies that highlight how humans do not appear to behave optimally [17 19]. However, OFC is best viewed as a normative model that identifies the best possible solution for a given control problem (like Bayesian inference for decisional processes). OFC crystalizes what good control ought to look like, generating a wealth of /$ see front matter ß 212 Elsevier Ltd. All rights reserved. Trends in Cognitive Sciences, November 212, Vol. 16, No

2 [(Figure_1)TD$FIG] Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 (a) Task selec on Op mal feedback control policy Motor commands Efference copy Op mal state es ma on Sensory feedback (b) Basal ganglia PF PP dpm vpm SMA A5 Controlled plant Spinal cord dpm vpm SMA A5 S1 M1 PP dpm A5 S1 Cerebellum Figure 1. Optimal Feedback Control and neural implementation. (a) Optimal Feedback Control Policy converts state variables to motor commands. Optimal State Estimation uses efference copy of motor commands and sensory feedback to estimate state variables. Task Selection specifies the goal of the behaviour which then defines the corresponding control policy. (b) Neural implementation of OFC-like voluntary control. The controlled plant is assumed to include the limb and spinal cord. The three basic processes, task selection, control policy, and state estimation, are each generated by highly distributed circuits. Dashed boxes denote individual cortical regions and some cortical regions are assumed to participate in two or even all three processes. Cortical connections are assumed amongst cortical regions involved in a given process. PF, PP and S1 include multiple subdivisions not displayed for simplicity. Cortical abbreviations: PF, prefrontal; PP, posterior parietal; dpm, dorsal premotor; vpm, ventral premotor; SMA, supplementary motor area; A5, area 5; S1, primary somatosensory; M1, primary motor. Images of spinal cord and limb adapted from [21]. experimental predictions on the general properties that biological controllers could or should possess. In fact, observed deviations from optimal behaviour provide extremely useful insight into the tricks and tradeoffs inherent in biological control. The mathematical tools used to solve optimal control problems are limited largely to linear control problems with some extensions for nonlinear systems [8]. Clearly, the brain does not use the formalisms of OFC to solve control problems. This is often used as a criticism of OFC [2], although these alternatives inevitably have the same problem. The assumption is that genetics and learning provide good solutions that are OFC-like. Perturbations as a probe of voluntary control Because feedback is an essential feature of OFC, a key approach to probe control is through perturbations, either visual or mechanical. The use of mechanical perturbations is not new this approach has been used for many decades to observe muscle stretch responses (Box 1). What OFC clarifies is that control is highly specific to the ongoing behavioural goal. Perturbations do not just elicit stereotypical motor reflexes, but also voluntary control processes directly [21]. Thus, perturbations provide an important window into voluntary control. Recent studies highlight several important features of these rapid corrective responses (see [22], for a detailed review). Behavioural goal One key feature of OFC is that errors are only corrected if they affect the behavioural goal and are ignored if they do not. This principle has been shown in a recent study that compared corrective responses when subjects reached to a circular target versus a rectangular bar (Figure 2, [23]). Corrective responses were directed back to the circular target, whereas responses for the rectangular bar were redirected to a new location along the bar. Subjects took advantage of the rectangular shape by making smaller corrective responses closer to the perturbed position of the hand. Differences in electromyographic (EMG) activity for corrective responses to the circular and rectangular targets were observed in as little as 7 ms. Further, obstacles in the environment altered these corrective responses. Long latency responses can also be altered to maximize overall success [24]. Blocks of predominantly resistive perturbations resulted in an increase in perturbation 542

3 Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 Box 1. Electromyographic responses to a perturbation A mechanical perturbation applied to a limb (or joint) elicits activity in muscle afferents (and likely some cutaneous afferents), generating a complex pattern of electromyographic (EMG) activity in muscles that resist the perturbation (Figure I; for a review, see [22]). This pattern of activity is generally divided into a number of phases. The earliest response occurs from 2 to 4 ms (varies somewhat dependent on distance of muscle from spinal cord), is called the short latency reflex, and is generated entirely by the spinal cord. Activity after 1ms has traditionally been viewed as voluntary, because movement-related EMG can be generated at this time from visual or somatosensory stimuli [4]. The time from 5 to 1 ms is termed the long latency response and involves both spinal and supraspinal feedback. Stretch responses to mechanical perturbations provide a simple paradigm to study feedback corrections during voluntary control. The location of muscle spindles (in muscles!) creates an inherent link between their activity and motor errors, whereas other sensory systems are only conditionally involved (which is also interesting for this very reason). The large diameter of primary muscle-spindle afferents makes transmission to the spinal cord relatively fast and suggests that the motor system has put a premium on their role in rapid feedback processing. There are several factors that need to be considered when exploring corrective responses: (i) OFC highlights how feedback depends on the behavioural goal, thus it is important to define a clear behavioural goal to the subject, both before and after the perturbation. (ii) The task-dependent flexibility of stretch responses impacts the types of perturbations one should apply. Motors are commonly used to apply perturbations but can be used in two different modes. In force control, the motor specifies a force and subjects can counter these perturbations in order to continue to perform a behavioural task. In motion control, the motor specifies a position (or velocity), making it impossible to attain a behavioural goal. Subjects may attempt to compensate, but this is difficult to measure experimentally. (iii) The stretch response can be quickly quenched 3 ms following the offset of a perturbation [69]. Thus, step perturbations that remain on to the end of the trial allow the full multi-phasic EMG response to be observed. (iv) Perturbation magnitude is a balance between large enough to elicit clear EMG responses, but small enough so that subjects can continue to perform the behavioural goal. This magnitude effect may explain why homonymous short-latency responses were observed in [44], whereas heteronymous responses were found for large perturbations in [31]. (v) Background loads prior to the perturbation allow excitatory and inhibitory responses to be observed. Short latency responses are also difficult to observe without some background muscle activity. [(Box_1)TD$FIG] Perturba on onset Applied step-torque perturba on EMG of stretched muscle Spinal propriocep ve feedback Short latency R1 5 Joint mo on 5 Cor cal propriocep ve feedback Long latency R2 1 1 R3 5 1 Time (ms) Visual feedback Voluntary Figure I. Mechanical perturbation applied to a joint causes joint motion and a multiphasic electromyographic response in stretched muscles. responses in order to maintain task success as compared to when such perturbations were rare. Shifts in visual feedback (visual perturbations) provide a parallel approach to study the use of vision for control. The influence of target shape was highlighted in a recent study that used visual shifts in hand position during reaching [25]. Corrections for rectangular targets oriented parallel to the movement path elicited larger corrective responses than corrections for targets oriented perpendicular to the movement path. Subjects can also ignore visual shifts in hand feedback, if required. Franklin and Wolpert [9] found that subjects made larger corrective responses when visual shifts remained to the end of the trial, whereas smaller corrective responses were elicited if the visual shift was only transiently present. Everyday activities require us to use each limb independently (e.g., use a spoon to eat cereal) or to coordinate the limbs together (e.g., carry a large bowl). As neural circuitry for controlling each limb is largely confined to separate halves of the brain, bimanual motor tasks provide an interesting model to study task-dependent feedback [26]. When the two limbs are used in a bimanual motor task, perturbations applied to one limb result in corrective responses in the other limb [27 29]. Impressively,EMG changes to the perturbation can be observed in less than 1 ms for these inter-limb corrections (Figure 3, [29 31]. The presence of goal-directed corrections touches on an age-old debate on whether there is a desired trajectory during tasks such as reaching. Alternative models to OFC commonly assume that the motor system computes a desired trajectory [2]. If the goal is to put the hand at a spatial goal, OFC suggests that there is no need for a desired trajectory. Although there appears to be some importance placed on initiating relatively straight movements [18],even in congenitally blind subjects [32], corrective responses do not return to a desired trajectory. Movements performed in a curl force field that tends to push the hand sideways result in over-compensation early in the movement [33]. Why should the brain produce a greater force than is necessary to overcome a perturbation at the start of a reaching movement? Optimal control predicts this pattern as early 543

4 [(Figure_2)TD$FIG] Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 (a) Hand paths y 2 cm x (b) Posi on (m) X posi on Time (ms) EMG (au) (c) Brachioradialis R 1 R 2 R Time (ms) Figure 2. Corrective responses to mechanical perturbations applied when reaching to a circle or rectangular bar. (a) Black lines are unperturbed reaching movements to each target. Display of rectangular bar in diagram is clipped on the left, because unperturbed reaches were made roughly to the centre of the bar. Extensor perturbations were applied to each joint just after movement onset. Corrective responses were directed back to the circular goal (blue lines), whereas corrective responses were redirected along the bar near the perturbed hand (red lines). (b), (c) Differences in hand corrections between target shapes can be observed in just over 2 ms, with EMG differences in an elbow flexor observable in 6 ms post-perturbation. Vertical lines denote perturbation onset and dashed lines denote separation of stretch response s. Adapted from [23]. compensation is advantageous for producing movements with minimum effort [23,33]. One could say that there is a nominal trajectory that is generated by the voluntary motor system if there were no noise or errors. However, any noise or error generates slight deviations from the nominal trajectory that leads to goal-directed corrections, creating a unique trajectory on each trial. In contrast, if the goal is to draw the letter h, [(Figure_3)TD$FIG] then the trajectory is the goal. Each individual has a unique way of writing the letters of the alphabet and this transfers into related control policies to generate that shape whether drawing using the hand, whole arm, or foot. Task selection The studies described above required subjects to counter a perturbation and continue to perform a motor task. However, many studies ask subjects to generate another task after the perturbation, such as giving instructions to (a) Brachioradialis Triceps longus Key: Contralateral flexion Contralateral extension (b) Time from perturba on onset (ms) Time from perturba on onset (ms) Figure 3. Inter-limb corrective responses elicited by perturbations to a single limb. Schematics show EMG recorded from the unperturbed right arm during perturbations of the left arm. (a) Group averages of normalized EMG in the uncoupled control task when the contralateral arm was perturbed either downward (blue traces) or upward (orange trace). No statistical change in the either muscle was observed in this task. (b) Response in the coupled single-object task when the two hands controlled a single bar and the subject was instructed to maintain it in a spatial target that was parallel to the body. Red vertical line indicates the earliest time at which differences in EMG were observed between the two conditions. Adapted from [29]. 544

5 Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 resist/let go after the perturbation. In effect, the perturbation elicits a change in the control policy, from maintaining a fixed posture to generating a rapid movement to oppose the load for resist or relaxation for let go. In order to remove the ambiguity of the behavioural goal inherent in these verbal instructions, Pruszynski et al. [34] introduced a spatial version of this task in which a perturbation pushedthehandeitherintoorawayfromaspatialgoal. Subjects can easily launch goal-directed motor actions throughout the workspace with corresponding EMG responses present in 6 ms. Such motor responses are not entirely preplanned (triggered), given that the response scales with the size of the perturbation [35]. Motor reflexes, such as the stumble corrective response or flexion/crossed-extension reflexes, are triggered responses that interrupt an ongoing motor pattern, such as walking [36]. These corrective responses are only observed for large perturbations or stimuli. Perturbations in the studies described above were all relatively large in order to generate clear EMG responses. Although subjects were still able to attain the behavioural goal, the question is whether these corrective responses are distinct from voluntary control, like the stumble corrective response. This issue was addressed by observing EMG responses to multi-joint perturbations of varying magnitude [37]. Significant muscle responses were observed, even when perturbations generated extremely small changes in angular motion that approached the natural range of variability observed during unperturbed trials. The use of linear regressions found that the scaling between perturbation magnitude and EMG activity passed through the origin. This suggests that the perturbations used in most studies cause corrective responses that reflect feedback processing that is also present during unperturbed voluntary motor actions. The ability of mechanical perturbations to launch impending motor plans provides an interesting way to probe decisional processes. For example, a recent study examined stretch responses as subjects judged the direction of motion in a random dot motion display [38]. The authors probed the state of the motor system by perturbing the arm at random times during decision formation and found that long-latency responses were modulated by the strength and duration of the motion pattern prior to their decision to respond to the stimuli. Long-latency responses can also be quickly updated to new spatial goals [39,4]. For example, long latency responses can be modified in as little as 1 ms following the presentation of a spatial goal [4]. Interestingly, responses to mechanical perturbations applied 1 ms after the shift in the target during reaching also result in a corresponding change in the corrective response, including modulation of short latency spinal responses [39]. Thus, shifts in the behavioural goal lead to a shift in movement direction and corresponding changes in feedback corrections, both now directed towards the new spatial goal. These studies highlight the intimate link between decisional processes and motor control [41,42]. Limb and environment The physics of limb movement are complex as the activity of a muscle spanning a joint can lead to motion at multiple joints. The voluntary motor system considers these interactions [43], but what about feedback corrections? This question was addressed by applying loads to the shoulder and elbow, yielding particular motion patterns at these two joints [44]. In one experiment, pure shoulder torque or pure elbow torque perturbations were applied, resulting in the same shoulder motion but different elbow motion. Perturbation-related activity of mono-articular shoulder muscles was examined to see whether they reflected the shoulder motion or the underlying shoulder torque. The short-latency response was the same in both conditions, indicating that it responded only to local joint motion. In contrast, the long-latency response considered the motion at both joints, generating a larger response for the shoulder torque perturbation than the elbow torque perturbation. In a second experiment, combined shoulder and elbow torque perturbations were applied generating no motion at the shoulder and either flexion or extension motion at the elbow (Figure 4a). If the stretch response of shoulder muscles account for the limb s mechanical properties, they should still respond to this perturbation, even though the joint is not moving. Again, short-latency response of the monoarticular shoulder extensor muscle (posterior deltoids) mirrored the local joint motion. Importantly, there was a robust long-latency response that appropriately countered the underlying torque perturbation (Figure 4b). Knowledge of the physics of the limb is also used for online corrections during reaching for mechanical perturbations [45], visual jumps in target location [46], and even motor planning [47]. The physical environment alters the forces necessary to move our bodies, whether it is an object we are holding or the viscous forces generated when moving underwater. There is a large literature demonstrating that the voluntary motor system can quickly adapt to changes in the visual or mechanical environment [4,48]. Evidence is now mounting that corrective responses are also altered during these adaptations. Burdet et al. [49] and others [5,51] imposed a diverging force field on reaching movements and found that, with practice, corrective responses during the reach increased along an axis that was parallel to the diverging field. When people reached in a curl field in which the forces pushed the hand in only one direction, perturbation responses increased and reflected the properties of the load [52 55]. If a perturbing force is present only for a portion of the movement, long latency responses are modified even before the hand enters that region [56,57]. Adaptation to novel loads result in changes in long latency responses prior to movement during the preparatory period [58]. If the force field perturbs the hand, but this perturbation is irrelevant to task success (i.e., the cursor moves to the target in any case), corrective responses are decreased during the reach [59]. Finally, visual feedback corrections are also modified when adapting to novel mechanical loads [6]. Neural implementation Spinal and transcortical feedback As a theory to describe behaviour, OFC is agnostic as to how voluntary motor control is generated by the spinal cord and brain. The spinal cord provides the first level of 545

6 [(Figure_4)TD$FIG] Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 (a) (c) t e t s t e 8 Excitatory shoulder torque (b) Perturba on onset t s (d) Perturba on onset No shoulder Mo on Inhibitory shoulder Torque x917a Post. deltoid EMG R1 R2/R3 1au Firing rate (Hz) Responding to underlying torque 5ms Time (ms) Figure 4. Long-latency response accounts for limb dynamics. (a) Subjects were instructed to maintain their hand at a central target and step-torque perturbations were randomly applied to both the shoulder and elbow (flexor torques at both joints, red, or extensor torques at both joints, blue). Shoulder and elbow torque magnitudes and arm geometry were chosen so that they caused substantial elbow motion but minimal shoulder motion. (b) Muscle activity aligned on perturbation onset. Note that even though the shoulder muscle was neither stretched nor shortened by the mechanical perturbation, there is still a robust long-latency response (both excitatory and inhibitory) in the shoulder extensor muscle. Adapted from [44]. (c), (d) Response of an exemplar shoulder-like M1 neuron (c) and the population of shoulder-like neurons (d) to a mechanical perturbation that causes pure elbow motion. Same color scheme as in (a). Note that the shoulder-like neurons respond to the underlying shoulder torque, even though the local motion information from the shoulder is ambiguous. Adapted from [71]. feedback processing. In many species, spinal circuits support sophisticated control, such as scratching, wiping, and basic locomotor patterns [36,61,62]. Analyses of these movements highlight many of the key characteristics predicted by OFC, notably success with variability. The spinal cord also provides phase-dependent feedback in humans during cyclical tasks such as locomotion and even hand cycling [63,64]. Recent work also highlights the importance of spinal circuits for transmitting descending commands during voluntary motor actions [65], although the specific computational processes provided at this level remains unknown. The short-latency stretch response is generated only by spinal feedback. Its size is relatively small and can only be modulated over days or weeks with training [66]. Shortlatency spinal responses are modified by background load due to the effects of motoneuron recruitment, but this is not necessarily a beneficial quality [67]. Sensory gating also occurs at the spinal cord prior to and during movement [68]. Stretch responses can be quickly halted 3 ms after perturbations are removed, demonstrating a powerful inhibitory capability of the spinal cord to quench stretch responses [69]. Although spinal feedback certainly contributes to longlatency responses, a transcortical feedback pathway starts to contribute at this time. Several studies have demonstrated that TMS used on primary motor cortex influences feedback corrections during the long-latency time period [54,56,7,71]. In non-human primates, many studies in the 197s and early 198s demonstrated how primary motor cortex (M1) neurons respond to mechanical perturbations or passive limb motion (for reviews, see [3,72]). A classic study by Evarts and Tanji [73] examined neural processing in M1 in a variant of the intervene/do not intervene task by training monkeys to respond to a mechanical perturbation by either pulling or pushing the perturbing handle. Early activity in M1 (2-5 ms post-perturbation) reflected the applied perturbation, but was quickly followed by the appropriate motor response at 5 ms. M1 neurons are directionally tuned to mechanical perturbations [74] in the same manner that activity is broadly tuned to various parameters of movement during reaching [75] or load-related activity during posture [76]. The impact of sensory feedback on M1 activity has also recently been demonstrated by comparing how encoding models of M1 activity depend on the ongoing movements of the limb [77]. In this study, an encoding model was generated from M1 activity when the monkey made reaching movements to spatial targets. Subsequent performance of the monkey on the use of the model to move a cursor to spatial targets was best when the monkey was also moving its hand to the spatial targets, but degraded somewhat when the limb remained stationary and was worst when the limb was randomly moved around the workspace. Thus, changes in 546

7 Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 afferent signals influenced ongoing activity in M1. The impact of sensory feedback on M1 de-emphasizes ideas of neural coding and emphasizes neural dynamics when interpreting M1 function [78]. A recent study examined whether the transcortical pathway through M1 considers the mechanical properties of the limb [71] using the paradigm developed by Kurtzer and colleagues [44]. Unlike muscles that have clear anatomical actions, the study identified shoulder and elbowrelated M1 neurons based on their response to steady-state motor outputs as the monkey countered various combinations of shoulder and elbow loads [74]. The earliest response, 2 to 5 ms after perturbation onset, did not distinguish between the various loading conditions, similar to the default response observed in Evarts and Tanji [73]. However, at 5ms the shoulder-related neurons rapidly increased their firing for the appropriate load at the shoulder, approximately 2 ms before shoulder muscles appropriately responded to the applied load (Figure 4c and d). Thus, in M1 (or elsewhere in this pathway), elbow and shoulder motion are integrated to identify the underlying muscular torque that will counter the applied load. Some models of voluntary control focus attention on spinal feedback [14,15]. However, the observation that virtually all sophisticated stretch responses occur during the long- and not the short-latency time period suggests that supraspinal feedback must be important for voluntary control. A shift to higher motor centres may have occurred as the complexity and breadth of motor skills increased through evolution. Spinal circuits may provide good control, but only for a limited range of behaviours, such as quadrupedal locomotion, scratching, and other stereotyped behaviours. As animals, notably primates, began to increase their breadth of motor skills, such as goal-directed reaching for objects in the environment and social grooming, and increased the use of vision for control, the voluntary motor system may have required a more general, highly-flexible approach to control, increasing the size and importance of corticocerebellar circuits. One reason for this shift to supraspinal control may reflect the importance of prediction in optimal state estimation, because the use of efference copies for control requires knowledge of the properties of the limb, contact forces, and prior experience (but see, [79]). Mapping OFC onto brain circuits OFC-like control is speculated as being generated by cortical and sub-cortical brain circuits [21,8]. Figure 1b provides an overview of the putative contributions of various brain regions to voluntary control. It is at best a sketch that will evolve as experimentation helps to delineate further the contribution of different brain regions to control. The three basic processes, task selection, control policy, and state estimation, are each generated by highly distributed cortical circuits. The basal ganglia participate in task selection through connections with associated cortical regions and the cerebellum participates in optimal state estimation through associated connections with cortex [81 83]. The diagram highlights that many cortical regions support each process and a given cortical region likely participates in two or even all three processes. Interactions between the three processes can occur between connected cortical regions within a given process or within a cortical region that is involved in multiple processes. There are also direct projections from basal ganglia and cerebellum to M1. From the perspective of supraspinal circuits, the spinal cord may be viewed as part of the peripheral motor apparatus modifying the force-length and force-velocity relationships of the limb, simplifying the properties of the musculoskeletal system [84]. More complex spinal processes certainly exist. However, it seems unlikely that all the complexities observed for spinal processing during cyclical behaviours of the hindlimb of non-primates are engaged during voluntary motor actions in the upper limb of humans. Regardless, the prediction is that supraspinal circuits recognize and compensate for spinal complexities much like they recognize and compensate for properties of the musculoskeletal system. The model emphasizes the distributed nature of feedback control much like decisional processes are presumed to involve both frontal and parietal circuits [42,85]. Transcortical feedback for stretch responses has been assumed to reflect a relatively simple pathway of sensory transmission from primary somatosensory cortex to primary motor cortex. However, state estimation is complex, involving the conversion of efferent copy signals into estimates of the present position of the limb, as well as integration of delayed sensory feedback from visual, muscle, and cutaneous afferents. These processes likely involve both cerebellar and parietal cortical regions. A key challenge for research is to understand how different feedback pathways contribute to state estimation during online control. Sensorimotor integration has been examined predominantly during motor planning before the initiation of movement. Does sensory integration observed prior to movement reflect how such processes occur during motor action? Are all signals integrated in a single representation that represents the present state of the system or is each signal processed individually and signals only converge in motor areas such as M1? Present models emphasize a serial organization across cortical regions with descending control principally generated by M1. However, M1 provides only 4% of the axons in the corticospinal tract [86]. Premotor and somatosensory cortex also directly influence spinal processing. Given that the spinal cord is viewed in the present model as part of the plant, one can consider operations such as modifying sensory processing in the spinal cord and modifications of gamma motoneuron activity as essential features of control. We need a broader view of control beyond the generation of EMG patterns. Concluding remarks The use of OFC to interpret voluntary movement has emphasized the importance of sensory feedback for control. Visual and mechanical perturbations during motor behaviour constitute an important scientific approach to probe the sophistication of feedback processing, providing a window into voluntary control processes. Recent studies reveal that corrective responses are highly adaptable based on the behavioural goal and consider the many complexities inherent in multi-joint movements. The basic computational 547

8 Review Trends in Cognitive Sciences November 212, Vol. 16, No. 11 Box 2. Questions for future research How is OFC-like control generated without the mathematical formalisms? The brain clearly does not use the mathematical formalisms of OFC and a major challenge is to understand how learning and evolution can generate OFC-like behavior. How is OFC-like control generated by the brain? Figure 1b identifies which brain regions may contribute to a specific process, but not how. State estimation is a complex process involving forward models to estimate the present state from efference copy and delayed sensory signals, and integration across these multiple sources of information. How do the many feedback pathways in the brain generate state estimation? Even a single sensory modality has multiple feedback pathways or processes [87]. What is the role of spinal feedback during voluntary control? Spinal feedback is clearly important in cyclical behaviours, but appears to be more limited during voluntary control. The ability to record spinal neurons in non-human primates during voluntary behaviour provides an important method to address this question [88,89]. Also important to consider is evidence of predictive signals in the firing patterns of muscle spindle afferents during human movement [9], which suggests that this sensory system provides more than just muscle length and velocity feedback. processes in OFC task selection, control policy and state estimation provide an interesting way to interpret the contributions of brain regions to voluntary control, guiding future research on the neural basis of voluntary motor action (Box 2). Acknowledgements This work was supported by grants from the National Science and Engineering Research Council of Canada, Canadian Institutes of Health Research (CIHR), and a GlaxoSmithKlein-CIHR chair in Neurosciences. The author would like to thank members of the Limb Lab for helpful feedback on this article. References 1 Sherrington, C.S. (191) Flexion-reflex of the limb, crossed extensionreflex, and reflex stepping and standing. J. Physiol. 4, Powers, W.T. (28) Living Control Systems III: The Fact of Control, Benchmark Publications 3 Scott, S.H. (28) Inconvenient truths about neural processing in primary motor cortex. J. Physiol. 586, Franklin, D.W. and Wolpert, D.M. (211) Computational mechanisms of sensorimotor control. Neuron 72, Todorov, E. and Jordan, M.I. (22) Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, Todorov, E. (26) Optimal control theory. In Bayesian Brain: Probabilistic Approaches to Neural Coding (Doya, K. et al., eds), pp , MIT Press 7 Faisal, A.A. et al. (28) Noise in the nervous system. Nat. Rev. Neurosci. 9, Li, W. and Todorov, E. (27) Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic systems. Int. J. Control 8, Franklin, D.W. and Wolpert, D.M. (28) Specificity of reflex adaptation for task-relevant variability. J. Neurosci. 28, Nagengast, A.J. et al. (29) Optimal control predicts human performance on objects with internal degrees of freedom. PLoS Comput. Biol. 5, e Valero-Cuevas, F.J. et al. (29) Structured variability of muscle activations supports the minimal intervention principle of motor control. J. Neurophysiol. 12, Stevenson, I.H. et al. (29) Bayesian integration and non-linear feedback control in a full-body motor task. PLoS Comput. Biol. 5, e Dingwell, J.B. et al. (21) Do humans optimally exploit redundancy to control step variability in walking? PLoS Comput. Biol. 6, e Feldman, A.G. and Levin, M.F. (29) The equilibrium-point hypothesis past, present and future. Adv. Exp. Med. Biol. 629, Loeb, G.E. (212) Optimal isn t good enough. Biol. Cybern. dx.doi.org/1.17/s Friston, K. (211) What is optimal about motor control? Neuron 72, Hudson, T.E. et al. (21) Compensation for changing motor uncertainty. PLoS Comput. Biol. 4, e Kistemaker, D.A. et al. (21) The central nervous system does not minimize energy cost in arm movements. J. Neurophysiol. 14, De Rugy, A. et al. (212) Muscle coordination is habitual rather than optimal. J. Neurosci. 23, Mohan, V. and Morasso, P. (211) Passive motion paradigm: an alternative to optimal control. Front. Neurobot. 5, 4 21 Scott, S.H. (24) Optimal feedback control and the neural basis of volitional motor control. Nat. Rev. Neurosci. 5, Pruszynski, J.A. and Scott, S.H. (212) Optimal feedback control and the long-latency stretch response. Exp. Brain Res. 218, Nashed, J.Y. et al. (212) Influence of the behavioural goal and environmental obstacles on rapid feedback responses. J. Neurophysiol. 18, Cheng, E.J. and Loeb, G.E. (28) On the use of musculoskeletal models to interpret motor control strategies from performance data. J. Neural Eng. 5, Knill, D.C. et al. (211) Flexible, task-dependent use of sensory feedback to control hand movements. J. Neurosci. 31, Diedrichsen, J. et al. (21) The coordination of movement: optimal feedback control and beyond. Trends Cogn. Sci. 14, Diedrichsen, J. (27) Optimal task-dependent changes of bimanual feedback control and adaptation. Curr. Biol. 17, Diedrichsen, J. and Gush, S. (29) Reversal of bimanual feedback responses with changes in task goal. J. Neurophysiol. 11, Dimitriou, M. et al. (212) Task-dependent coordination of rapid bimanual motor responses. J. Neurophysiol. 17, Mutha, P.K. and Sainburg, R.L. (29) Shared bimanual tasks elicit bimanual reflexes during movement. J. Neurophysiol. 12, Manning, C.D. et al. (212) Proprioceptive reaction times and longlatency reflexes in humans. Exp. Brain Res. 221, Sergio, L.H. and Scott, S.H. (1998) Hand and joint paths during reaching movements with and without vision. Exp. Brain Res. 122, Izawa, J. et al. (28) Motor adaptation as a process of reoptimization. J. Neurosci. 28, Pruszynski, J.A. et al. (28) Rapid motor responses are appropriately tuned to the metrics of a visuospatial task. J. Neurophysiol. 1, Pruszynski, J.A. et al. (211) The long-latency reflex is composed of at least two functionally independent processes. J. Neurophysiol. 16, Rossignol, S. et al. (26) Dynamic sensorimotor interactions in locomotion. Physiol. Rev. 86, Crevecoeur, F. et al. (212) Fast corrective responses are evoked by perturbations approaching the natural variability of posture and movement tasks. J. Neurosphysiol. 17, Selen, L.P.J. et al. (212) Deliberation in the motor system: reflex gains track evolving evidence leading to a decision. J. Neurosci. 32, Mutha, P.K. et al. (28) Visual modulation of proprioceptive reflexes during movement. Brain Res. 1246, Yang, L. et al. (211) Rapid motor responses quickly integrate visuospatial task constraints. Exp. Brain Res. 211, Wolpert, D.M. and Landy, M.S. (212) Motor control is decision-making. Curr. Opin. Neurobiol Cisek, P. (212) Making decisions through a distributed consensus. Curr. Opin. Neurobiol Gribble, P.L. and Ostry, D.J. (1999) Compensation for interaction torques during single- and multijoint limb movements. J. Neurophysiol. 82, Kurtzer, I.L. et al. (28) Long-latency reflexes of the human arm reflect an internal model of limb dynamics. Curr. Biol. 18,

9 Review Trends in Cognitive Sciences November 212, Vol. 16, No Kurtzer, I. et al. (29) Long-latency responses during reaching account for the mechanical interaction between the shoulder and elbow joints. J. Neurophysiol. 12, Gritsenko, V. et al. (29) Integration of predictive feedforward and sensory feedback signals for online control of visually guided movement. J. Neurophysiol. 12, Cos, I. et al. (211) The influence of predicted arm biomechanics on decision making. J. Neurophysiol. 15, Shmeulof, L. and Krakauer, J.W. (211) Are we ready for a natural history of motor learning? Neuron 72, Burdet, E. et al. (21) The central nervous system stabilizes unstable dynamics by learning optimal impedance. Nature 414, Franklin, D.W. et al. (27) Endpoint stiffness of the arm is directionally tuned to instability in the environment. J. Neurosci. 27, Wong, J. et al. (29) The influence of visual perturbations on the neural control of limb stiffness. J. Neurophysiol. 11, Wang, T. et al. (21) Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses. Biol. Cybern. 85, Wagner, M.J. and Smith, M.A. (28) Shared internal models for feedforward and feedback control. J. Neurosci. 28, Shemmell, J. et al. (29) The differential role of motor cortex in stretch reflex modulation induced by changes in environmental mechanics and verbal instruction. J. Neurosci. 29, Krutky, M.A. et al. (21) Interactions between limb and environmental mechanics influence stretch reflex sensitivity in the human arm. J. Neurophysiol. 13, Kimura, T. et al. (26) Transcranial magnetic stimulation over sensorimotor cortex disrupts anticipatory reflex gain modulation for skilled action. J. Neurosci. 26, Kimura, T. and Gomi, H. (29) Temporal development of anticipatory reflex modulation to dynamical interactions during arm movement. J. Neurophysiol. 12, Ahmadi-Pajouh, M.A. et al. (212) Preparing to reach: selecting an adaptive long-latency feedback controller. J. Neurosci. 32, Scheidt, R.A. et al. (25) Interaction of visual and proprioceptive feedback during adaptation of human reaching movements. J. Neurophysiol. 93, Franklin, S. et al. (212) Visuomotor feedback gains upregulate during the learning of novel dynamics. J. Neurophysiol. 18, Stein, P.S. (25) Neuronal control of turtle hindlimb motor rhythms. J. Comp. Physiol. A 191, Frigon, A. (212) Central pattern generators of the mammalian spinal cord. Neuroscientist 18, Zehr, E.P. and Duysens, J. (24) Regulation of arm and leg movement during human locomotion. Neuroscientist 1, Zehr, E.P. et al. (212) Neural control of rhythmic arm cycling after stroke. J. Neurophysiol. 18, Alstermark, B. and Isa, T. (212) Circuits for skilled reaching and grasping. Annu. Rev. Neurosci. 35, Wolpaw, J.R. (21) What can the spinal cord teach us about learning and memory? Neuroscientist 16, Pruszynski, J.A. et al. (29) Temporal evolution of Automatic Gain- Scaling. J. Neurophysiol. 12, Seki, K. and Fetz, E.E. (212) Gating of sensory input at spinal and cortical levels during preparation and execution of voluntary movement. J. Neurosci. 32, Kurtzer, I. et al. (21) Long-latency and voluntary responses to an arm displacement can be rapidly attenuated by perturbation offset. J. Neurophysiol. 13, Perreault, E.J. et al. (28) Interactions with compliant loads alter stretch reflex gains but not intermuscular coordination. J. Neurophysiol. 99, Pruszynski, J.A. et al. (211) Primary motor cortex underlies multijoint integration for fast feedback control. Nature 478, Hatsopoulos, N.G. and Suminski, A.J. (211) Sensing with the motor cortex. Neuron 72, Evarts, E.V. and Tanji, J. (1976) Reflex and intended responses in motor cortex pyramidal tract neurons of monkey. J. Neurophysiol. 39, Herter, T.M. et al. (29) Comparison of neural responses in primary motor cortex to transient and continuous loads during posture. J. Neurophysiol. 11, Kalaska, J.F. (29) From intention to action: motor cortex and the control of reaching movements. Adv. Exp. Med. Biol. 629, Herter, T.M. et al. (27) Characterization of torque-related activity in primary motor cortex during a multi-joint postural task. J. Neurophysiol. 97, Suminski, A.J. et al. (21) Incorporating feedback from multiple sensory modalities enhances brainmachine interface control. J. Neurosci. 3, Churchland, M.M. et al. (212) Neural population dynamics during reaching. Nature 487, Hantman, A.W. and Jessell, T.M. (21) Clarke s column neurons as the focus of a corticospinal corollary circuit. Nat. Neurosci. 13, Shadmehr, R. and Krakauer, J.W. (28) A computational neuroanatomy for motor control. Exp. Brain Res. 185, Strick, P.L. et al. (29) Cerebellum and nonmotor function. Annu. Rev. Neurosci. 32, Bostan, A.C. and Strick, P.L. (21) The cerebellum and basal ganglia are interconnected. Neuropsychol. Rev. 2, Prevosto, V. et al. (21) Cerebellar inputs to intraparietal cortex areas LIP and MIP: functional frameworks for adaptive control of eye movements, reaching, and arm/eye/head movement coordination. Cereb. Cortex 2, Nichols, T.R. and Houk, J.C. (1976) Improvement in linearity and regulation of stiffness that results from actions of stretch reflexes. J. Neurophysiol. 39, Cisek, P. and Kalaska, J.F. (21) Neural mechanisms for interacting with a world full of action choices. Annu. Rev. Neurosci. 33, Porter, R. and Lemon, R.N. (1993) Corticospinal Function and Voluntary Movement, Oxford University Press 87 Gritsenko, V. and Kalaska, J.F. (21) Rapid online correction is selectively suppressed during movement with a visuomotor transformation. J. Neurophysiol. 14, Takei, T. and Seki, K. (21) Spinal interneurons facilitate coactivation of hand muscles during a precision grip task in monkeys. J. Neurosci. 3, Shalit, U. et al. (212) Descending systems translate transient cortical commands into a sustained muscle activation signal. Cereb. Cortex 22, Dimitriou, M. and Edin, B.B. (21) Human muscle spindles act as forward sensory models. Curr. Biol. 2,