Parallel specification of competing sensorimotor control policies for alternative action options

Transcription

1 Parallel specification of competing sensorimotor control policies for alternative action options Jason P Gallivan 1, Lindsey Logan 4, Daniel M Wolpert 5 & J Randall Flanagan 1, npg 216 Nature America, Inc. All rights reserved. Recent theory proposes that the brain, when confronted with several action possibilities, prepares multiple competing movements before deciding among them. Psychophysical supporting evidence for this idea comes from the observation that when reaching towards multiple potential targets, the initial movement is directed towards the average location of the targets, consistent with multiple prepared reaches being executed simultaneously. However, reach planning involves far more than specifying movement direction; it requires the specification of a sensorimotor control policy that sets feedback gains shaping how the motor system responds to errors induced by noise or external perturbations. Here we found that, when a subject is reaching towards multiple potential targets, the feedback gain corresponds to an average of the gains specified when reaching to each target presented alone. Our findings provide evidence that the brain, when presented with multiple action options, computes multiple competing sensorimotor control policies in parallel before implementing one of them. In natural environments, we are often faced with a multitude of action possibilities. In such situations, it has been suggested that the brain prepares, in parallel, multiple potential movements before deciding between and implementing one of them 1,2. This compelling idea referred to as the affordance competition hypothesis receives empirical support from neurophysiological investigations in sensorimotor areas of the brain showing the parallel encoding of multiple potential reach, grasp and saccade targets before an animal decides between, and then makes a corresponding movement toward, one of these targets 1,4,5. Psychophysical support for this framework has come from spatial averaging behavior, whereby individuals, when required to launch a reach or saccadic eye movement before knowing which of several potential targets will be selected (that is, go-before-you-know), are found to aim their initial movements toward the midpoint of the target distribution, consistent with an averaging of the multiple competing movement directions 6 8. Recent evidence has further indicated that, in addition to movement direction, the reach paths 9 and hand orientations 1 required to contact the multiple competing targets may also be averaged. Taken together, the available neural and behavioral evidence suggests that the brain, when presented with multiple available actions, directly maps, in parallel, each of the competing options into associated motor variables. Although basic movement related variables such as direction, path and orientation can provide a useful characterization of some aspects of motor planning, it constitutes only a fraction of the actual parameterizations required for skilled behavior. A major component of any fully elaborated movement plan is the specification of a sensorimotor control policy, which refers to the setting of feedback gains that determine how the movement evolves as a function of the state of the motor system and how the motor system handles errors that can occur as a result of noise or external perturbations Indeed, according to several contemporary models of motor control, the initial movement trajectory is not explicitly planned, but rather emerges from the specification of a movement goal and feedback gains 16. Feedback gains have been shown to be flexibly adapted in accordance with the features of the task and environment and, according to the theory of optimal feedback control (OFC), are governed by a policy of minimum intervention, whereby the sensorimotor system responds strongly to errors that endanger the goal(s) of the task, but less vigorously to those that do not 1,14,22. Consistent with the predictions of the OFC framework, feedback gains, in response to either mechanical or visual perturbations, are heightened when reaching toward a narrower compared with a wider target 2,24 and can be rapidly updated (~1 ms), within a single movement, on the basis of a change in task parameters (for example, when the target jumps to a new location) 25. Given the clear importance of sensorimotor control policies in movement planning and control, the affordance competition hypothesis might predict that feedback gains should be specified for each of the alternative movement plans formed for each potential reach target. If so, then just as the initial movement trajectory in go-before-you-know tasks may reflect an average of the reach directions to the potential targets, the feedback gains should also reflect an average of the gains specified for each potential target. By measuring participants feedback gains while they performed reaches toward multiple potential targets, we tested this prediction of the parallel encoding framework. In our task, participants performed target-directed reaches toward two competing, superimposed targets of different widths (one narrow and one wide) and we varied the time, relative to reach onset, at which the final target was cued. We explored how visuomotor feedback 1 Department of Psychology, Queen s University, Kingston, Ontario, Canada. 2 Department of Biomedical and Molecular Sciences, Queen s University, Kingston, Ontario, Canada. Centre for Neuroscience Studies, Queen s University, Kingston, Ontario, Canada. 4 Life Sciences Program, Queen s University, Kingston, Ontario, Canada. 5 Department of Engineering, University of Cambridge, Cambridge, UK. Correspondence should be addressed to J.P.G. (gallivan@queensu.ca) or J.R.F. (flanagan@queensu.ca). Received 2 November 215; accepted December 215; published online 11 January 216; doi:1.18/nn VOLUME 19 NUMBER 2 FEBRUARY 216 nature NEUROSCIENCE

2 npg 216 Nature America, Inc. All rights reserved. gains, measured via the magnitude of involuntary rapid corrective responses to rapid shifts in the visual position of the hand halfway through the movement, were modulated as a function of uncertainty about target width during the reach. On the majority of trials, the final target was selected before or immediately after the reach was initiated (single-target and early-selection trials), providing participants with full certainty about the final target width very early into the movement. Notably, on the remaining trials, the final target was selected only near the very end of the movement (late-selection trials), after the corrective reflex response under the situation of target uncertainty (that is, when each target was equally likely) had already been probed. We found that the feedback gains for reaches toward multiple competing targets closely resembled an average of the gains on trials in which the final target (narrow or wide) was known very early into the movement (early-selection trials). This was despite the fact that participants, on late-selection trials, could just have easily performed the task by implementing the corrective responses associated with the narrow target, as this default control policy would have always ensured task success (given the overlapping potential targets). Rather, these results are consistent with the notion that, when presented with multiple action options, individuals prepare competing sensorimotor control policies in advance of selecting between them. Thus, our findings suggest that planning under conditions of target uncertainty involves simultaneously representing, for each potential movement, complex movement parameters beyond kinematics alone. RESULTS Our task required participants to perform target-directed reaches by moving the handle of a robotic manipulandum (Fig. 1a) that controlled the position of a cursor on a screen to contact one of two potential superimposed targets, one narrow and the other wide, that shared the same centroid. A countdown procedure following target presentation (five auditory beeps, each spaced 6 ms apart) provided the instruction for participants to prepare (beeps 1 ), move (beep 4) and then arrive at the final target (beep 5). Targets were either cued (filled-in) before movement onset (single-target trials) or at two different points (early or late) after the movement had been launched (two-target trials). Notably, on some trials, the visual position of the cursor on the screen was perturbed midway through the movement (under a visual occluder), requiring a rapid corrective response by participants to contact the target (Fig. 1b). To assess participants feedback gains in a manner uncontaminated by limb dynamics, we incorporated a force channel on a minority of these trials (Fig. 1c), which mechanically constrained the handle (and thus the hand) to a direct path between the start location and target 22,26. By measuring the corrective forces generated on these channel trials, we were able to probe the visuomotor feedback gains that participants adopted to deal with cases in which the final target was known (in one-target and early-selection two-target trials) versus unknown (in late-selection two-target trials) at the time of the cursor perturbation. Voluntary corrective responses across one- and two-target trials The cursor paths of a representative participant demonstrate the appropriate corrective responses, on non-channel trials, for cases when the cursor was perturbed to the left or right beneath the occluder (as well as the absence of these corrective responses on trials in which the cursor was not perturbed; Fig. 2a). These reach paths also show that, as expected, the movement corrections implemented on the narrow target trials are greater than those implemented on the wide target trials, particularly on the single and early target selection trials (see differences in reach traces at the final target locations). This latter observation becomes clearer when examining the reach endpoint distributions across these different target selection conditions (Fig. 2b,c). In the single-target trials for this participant, we observed more lateral (x) variation in the reach endpoint distribution for the wide target compared to the narrow target (Fig. 2b), consistent with previous results 2,24. In addition, as the duration of target uncertainty increased (from single to early-selection to late-selection trials), the lateral endpoint variability decreased for the wide target but increased slightly for the narrow target (Fig. 2b). To quantify lateral endpoint variability, we computed, for each participant, target selection condition and target width, the difference between the mean x endpoint positions for the leftward and rightward cursor jumps. We found a significant effect of target selection condition (single, early and late) Figure 1 Experimental methods. (a) Experimental setup. Participants (N = 8) were seated in a chair and grasped, with their right hand, the handle of a robotic manipulandum (vbot) that controlled the position of a cursor on a monitor. Visual feedback from the monitor was viewed through a mirror located in the plane of movement that prevented view of the participant s arm. (b,c) Examples of experimental conditions. On a subset of trials, the cursor was visually perturbed (to the left or right; right perturbation shown) exactly halfway through the movement (125 mm), after it passed under an occluder (gray horizontal bar). In the non-channel condition (b), participants had to use the handle to correct the position of the cursor to reach the target. In the force channel condition (c), which constrained the participants movement to a straight ahead path (denoted by dashed black vertical lines), the cursor position automatically jumped back after 25 ms. In b and c, the black and green boxes show the positions of the narrow and wide targets, respectively. Target selection (filling-in) could either occur before movement onset (single-target trials), early in the movement (at 25-mm y position; early-selection trials) or late in the movement (at 225-mm y position; late-selection trials). The average movement durations (across participants and conditions) associated with different epochs of the trial are displayed on the right. Note that the x length of the occluder has been shortened for display purposes. All other dimensions are drawn exactly to scale. nature NEUROSCIENCE VOLUME 19 NUMBER 2 FEBRUARY

3 Figure 2 Hand paths on non-channel trials and movement endpoint variance for each target selection condition. (a) Averaged trajectory traces for a representative participant. Left, single-target condition. Black rectangle and trajectory traces represent the narrow target (2 mm wide) and the associated movement paths to that target, respectively. Green rectangle and trajectory traces represent the wide target (8 mm wide) and the associated movement paths to that target, respectively. Gray horizontal bar represents the occluder. Middle, early-selection condition. In these trials, target selection (filling-in) occurred when the participant s cursor passed 25 mm of reach distance. Right, late-selection condition. In these trials, target selection occurred when the participant s cursor passed 225 mm of reach distance. (b) Reach endpoints for the participant shown in a plotted as a function of robot x-y position for each of the three experimental conditions. Endpoints are color-coded according to cursor jump direction (left and right directions, in red and dark blue, respectively, and no jump, in light blue). (c) Mean group (N = 8) differences in reach endpoint, plotted as a function of differences in mean x position (mean x position of the rightward perturbations minus the mean x position of the leftward perturbations), for each of the three experimental conditions. The lines represent means from individual participants and error bars denote ± s.e.m. *P <.5. npg 216 Nature America, Inc. All rights reserved. on the mean endpoint positions (F 2, 14 = 88.21, P <.1), as well as a significant effect for target width (narrow and wide; F 1, 7 = , P <.1). We also found a significant interaction between target width and selection condition on endpoint variability (F 2, 14 = 16.28, P <.1). The difference in variability between the narrow and wide targets was significant for each of the three conditions (Bonferroni corrected t tests; single: t 7 = 19.4, P <.1; early: t 7 = 11.1, P <.1; late: t 7 = 4.1, P =.9; Fig. 2c) even though the difference was substantially smaller in the late-selection trials than in the single and early-selection trials. Likewise, there was a significant effect of selection condition for each target width (Bonferroni corrected ANOVAs; narrow: F 2, 14 = 5.19, P =.42; wide: F 2, 14 = 161.9, P <.1). Variability was markedly less in the late-selection condition for the wide target, but slightly greater for the narrow target. Rapid involuntary corrective responses across one- and two-target trials Consistent with previous work 22,24, we found that, when we examined the forces exerted by the handle in channel trials over a 18 2-ms time window following the perturbation, the force in unperturbed trials was near zero, whereas the forces produced in response to the cursor perturbations were in the appropriate direction (that is, counteracting cursor displacement; Fig. ). Moreover, we observed that the corrective responses in the single-target and early-selection trials appropriately scaled with target size (that is, larger for narrow targets and smaller for wide targets, see refs. 2,24). Consistent with past work 22,25, we further found that the forces began showing reliable modulation approximately ms after the cursor perturbation onset, validating our use of the 18 2-ms time window for analysis. To be able to combine force responses in leftward and rightward perturbations trials, we inverted the measured force for the latter so that corrective forces were always positive. To test for effects at the group level, we computed, for each target width and selection condition, the average of each individual s median corrective forces for the (leftward) and + mm (rightward) perturbations (Fig. 4a). To investigate the effects of different aspects of target uncertainty, we carried out two targeted two-way repeated-measures ANOVAs. We first focused on the single-target and early-selection trials to assess the influence of initial target uncertainty (present on early-selection trials, but not single-target trials) under conditions in which participants had ample time to adjust feedback gains before perturbation onset. We then focused on the early-selection and late-selection trials to directly assess the influence of multiple competing targets at the time of the perturbation (present in late-selection trials, but not earlyselection trials) while equating for any residual effects of initial target uncertainly (present in both the late- and early-selection trials). Single-target versus early-selection conditions A 2 (single-target and early-selection) 2 (narrow and wide target) repeated-measures ANOVA revealed main effects of selection condition (F 1, 7 = 5.41, P =.2) and target type (F 1, 7 = 7.69, P <.1) on the corrective forces. Notably, however, the interaction between these factors was not significant (F 1, 7 = 2.191, P =.182). We followed up the results of this ANOVA using paired sample twotailed t tests. As expected, we found a significant difference in the 22 VOLUME 19 NUMBER 2 FEBRUARY 216 nature neuroscience

4 Figure Scaling of feedback gains (on channel trials) across target selection conditions. (a) Force readings, plotted as a function of time relative to cursor perturbation, for all trials and all conditions for a single representative participant. ms denotes the onset of cursor perturbation. Shaded gray vertical bars denote 18 2 ms post-perturbation, the time window reflecting the involuntary feedback response. The mean robot force value over this time window was extracted for each trial type and participant, and the median of these mean values were used for group-level analyses. (b) Median forces for the 18 2-ms time window and same participant shown in a. Force traces in a and open circles in b are color-coded according to the perturbation condition (light blue, unperturbed; dark blue and red, right and left perturbation, respectively). The color of the lines joining the open circles indicates the associated target size (green, wide target; black, narrow target). (c) Mean of median forces across participants (N = 8), plotted the same as in b. Small vertical lines in the open circles denote ± s.e.m. a Narrow target Force (N) Single-target trials Two-target trials Early selection Late selection Jump + mm Jump mm Jump mm npg 216 Nature America, Inc. All rights reserved. corrective forces associated with the narrow and wide targets in both the single-target (t 7 = 7.1, P =.2) and early-selection (t 7 = 6.68, P =.) conditions (Fig. 4a). These results indicate that, although participants maintained the same general sensitivity in their handling of the visual perturbation with regards to narrow versus wide targets, their overall feedback gains were upregulated in early-selection trials compared with singletarget trials (Fig. 4a). This suggests that the initial target uncertainty present on early-selection trials had a persistent effect on shaping participants rapid corrective responses, even though there was adequate time between the target cue and visual perturbation (>1 ms) 25 for individuals to adapt their reflex gains (Online Methods). Early versus late-selection conditions A 2 (early and late-selection) 2 (narrow and wide target) repeatedmeasures ANOVA did not reveal a main effect of selection condition (F 1, 7 =.467, P =.15) on corrective force, but did reveal a main effect of target size (narrow or wide; F 1, 7 =.768; P =.1), as well as a significant interaction (F 1, 7 =.285, P =.1). This suggests that, although participants displayed sensitivity to target size in their feedback gains on early-selection trials (as noted in the paired sample t tests performed above), this was not the case on the late-selection trials. A paired sample t test between the corrective forces implemented on narrow versus wide targets on late-selection trials confirmed that the forces generated did not show sensitivity to target width (t 7 =.61, P =.561). This finding is to be expected given that the feedback gains on these late-selection trials were measured in a time window (18 2 ms post-perturbation) before the target had actually been cued. Averaging of feedback gains in late-selection trials If participants generated an averaged feedback gain under conditions of target uncertainty, then we would expect the gain in late-selection trials to be intermediate between the gains for the wide and narrow targets in early-selection trials. To test this, we compared, using paired t tests, the average corrective force generated on late-selection trials (that is, the mean of responses to the narrow and wide target, where a difference was neither expected nor observed) with those generated on each of the early-selection trials (that is, for each the narrow and wide target). We chose the early-selection trials as the basis for our comparison given the prominent effect of initial target uncertainty (that is, the overall upregulation of feedback gains) observed for the early-selection compared with single-target selection conditions (noted above). Notably, this analysis showed that the corrective forces produced on the late-selection trials were significantly different from those associated with both the wide (t 7 = 2.62, P =.4) and narrow (t 7 = 5.1, P =.1) targets on early-selection trials. Thus, the Wide target b Average force between 18 and 2 ms (N) c Average force between 18 and 2 ms (N) Force (N) Time relative to perturbation (ms) Single-target trials Early selection Two-target trials Late selection Cursor jump (mm) Cursor jump (mm) visuomotor feedback gains observed on the late-selection trials, when the final target remains uncertain at the time of perturbation, lie somewhere in between those observed on the early-selection trials, when there is full target certainty at the time of perturbation and ample time to adjust the gains (Fig. 4a). In light of previous evidence suggesting that individuals, when presented with multiple competing reach targets and required to act before knowing the final target location, launch initial reaches that correspond to an average of the movement directions toward each target individually 6,7,9,1,27 29, we further explored whether individuals on the late-selection trials also implement feedback gains that resemble the average of those generated on the early-selection trials. To directly test this idea, we computed for each participant, from the early-selection trials, a synthetic average force the force that would be expected had participants explicitly averaged across the reflex gains nature NEUROSCIENCE VOLUME 19 NUMBER 2 FEBRUARY 216 2

5 npg 216 Nature America, Inc. All rights reserved. Figure 4 Group-level analysis of feedback gains. (a) Corrective forces (whereby forces from +-mm perturbations are multiplied by 1), averaged across participants, for the data shown in Figure c. The lines represent medians from individual participants and error bars denote ± s.e.m. *P <.5; n.s., non-significant (P =.561) (b) Relationship between mean force on late-selection trials (x axis, from rightmost plot in a) and early-selection trials (y axis, from middle plot in a). Filled black data points denote an average of the force response on narrow (open black points) and wide (open green points) target early-selection trials. Different symbols represent separate participants. Dashed line represents the identity line. (c) Corrective force as a function of trial block (averaged across bins of five trials) for early and late-selection trials with each target. associated with the narrow and wide potential targets and then examined how this synthetic average force relates to the mean force (averaged across narrow and wide target trials) implemented on lateselection trials. We derived this synthetic mean force by computing the average of participants forces on narrow and wide target earlyselection trials. Notably, for the majority of participants, these dataderived synthetic average force values fell along the identity line, and the average absolute deviation from the identity line was smaller for this data-derived force (M =.16 N) than for the force generated on either narrow (M =.1 N) or wide (M =.56 N) targets from the early-selection trials (Fig. 4b). When considering the signed deviations from the identity line, t tests revealed that only the deviations of the synthetic average force values did not significantly differ from zero (synthetic average: t 7 = 1.862, P =.15; narrow: t 7 = 2.621, P =.4; wide: t 7 = 5.96, P =.1). These findings suggest that, on late-selection trials, participants average across the reflex gains that are separately computed for the narrow and wide targets on earlyselection trials. Feedback gain averaging occurs at the onset of testing To what extent was this average feedback gain on late-selection trials something that was implemented immediately at the outset of the experiment and consistently, on a trial-by-trial fashion, rather than something that was gradually learned over the course of testing? When we examined the corrective force as a function of trial block (that is, a binned average of five trials), we found that the gain of the corrective response in late-selection trials was, right from the very beginning of the experiment, consistently intermediate between the gains for the narrow and wide targets in the early-selection trials (Fig. 4c). This indicates that the averaging of feedback gains was not some strategy that participants learned to implement over the course of the experiment. DISCUSSION Here we explored how individuals regulate their visuomotor feedback gains under conditions of target uncertainty. Notably, rather than using changes in task parameters (for example, shifts in target location) to introduce target uncertainty on a trial-by-trial fashion 25,, we instituted this uncertainty by presenting participants with multiple potential targets and manipulating the time at which one was cued. We report two main findings. First, we found that, on trials in which the final target information was not given until very early in the reach (that is, early-selection trials), individuals, despite showing similar target sensitivity to single-target trials, nevertheless exhibited an overall upregulation in their feedback gains. This residual and relatively long-lasting (>1 ms) effect of target uncertainty on reflex gains has not, to our knowledge, been described elsewhere. Second, we found that, on trials in which the final target was selected only after the visual cursor had been perturbed (that is, late-selection trials), individuals a Average corrective force between 18 and 2 ms (N) b Force on early-selection trials (N) c Average corrective force between 18 and 2 ms (N) Single-target trials * 2 1 Two-target trials Early selection Late selection * n.s. Interaction: * Narrow Wide Narrow Wide Narrow Wide 1 2 Average force on late-selection trials (N) (mean of narrow and wide targets) Narrow target Wide target Average of narrow and wide targets exhibited feedback gains that were well approximated by the average of the feedback gains implemented toward each of those targets on the early-selection trials. This finding, in light of previous neural and behavioral evidence showing that individuals prepare multiple competing reach movements 1,1 and average across their spatial directions when executing movements toward potential targets 6,9, suggests that motor averaging may constitute part of a general mechanism that the brain utilizes when dealing with environmental uncertainty. Residual effects of target uncertainty on feedback gains Previous studies have shown that the visual presentation of stimuli 2, as well as visual shifts in background,4, target location 5,6 and representation of hand position 7,8, elicit rapid motor responses. Notably, corrective movements in response to these visual displacements (for example, changes in the position representation of the hand or target) do not require that participants be consciously aware of such changes 5,9. This is consistent with the suggestion that the early components of these visually induced reflexive motor responses rely on involuntary rather than voluntary mechanisms 22,,4,6. Despite their involuntary nature, however, previous work has shown * * Trial block (5 trials per block) Early selection Narrow Wide Late selection Narrow Wide 24 VOLUME 19 NUMBER 2 FEBRUARY 216 nature neuroscience

6 npg 216 Nature America, Inc. All rights reserved. that these rapid corrective responses can be flexibly specified before movement onset, in accordance with task goals 19 22, and gradually updated based on the learnt dynamics of an environment 18. Although few studies to date have examined modulations of these rapid corrective motor responses in the context of a single goal-directed movement, one study 25 found that the CNS can intelligently modify its feedback responses, based on jumps in the location of a reach target, within 1 ms. In light of this evidence, it is notable, although perhaps not surprising, that participants in our early-selection trials (in which the target was cued, on average, 19 ms before the perturbation) were able to appropriately up- or downregulate their feedback gains in accordance with the width of the cued target (to a level of sensitivity observed on single-target trials). What is less clear is why, despite showing this goal-related sensitivity, participants also exhibited an overall upregulation in their reflex gains on these trials. Previous work has suggested that the optimal response to increased uncertainty is to decrease one s feedback gains,4,41. However, in these previous studies task uncertainty was introduced through sensory noise (for example, visual uncertainty of hand position or target location), and thus there is no advantage to setting up robust reflexive responses when the location of the target is unknown. Here, we introduced task uncertainty by presenting multiple potential targets of different widths, both of which were fully known in advance of movement 1. Thus, uncertainty was a result of participants lack of knowledge about which target would be selected, rather than in the sensory processing of those targets. This is an important distinction, as the optimal setting of feedback gains depends on where the uncertainty exists in a given system 42. If uncertainty is coupled to the sensory system, then the optimal control policy will be to reduce feedback responses 18. However, if the uncertainty is coupled to the external world (as in our task), or in one s own model of that world (that is, the internal model), then the optimal response will be to increase the feedback gains of the system 22 (and increase co-contraction, see ref. 4) while also reducing the contributions of predictive control 44. In our experiment, the upregulation of feedback gains on early-selection trials would be consistent with this optimal response. Nevertheless, given previous results 25, we find it surprising that participants still exhibited heightened reflex gains so late into the movement after target cuing (that is, ~19 ms). This indicates that the uncertainty introduced by multiple competing targets may have a much longer residual effect on the adjustment of reflex gains than has been previously demonstrated. Evidence for the parallel encoding of competing motor plans Mounting evidence suggests that, in situations affording several possible actions, multiple potential movements are prepared in parallel. Neural recordings from brain areas involved in eye movements 45, as well as areas involved in arm movements 1,2, have shown the simultaneous encoding of multiple competing targets before the decision to make an eye or reach movement, respectively, toward one of those locations. Consistent with these observations, psychophysical studies have shown that, when individuals are required to initiate an eye (for review, see ref. 8) or reach movement 46,47 before knowing which of several potential targets will be cued, their initial movement vector corresponds to a spatial average of the movements performed toward each target separately. Together, these findings suggest that a basic mechanism by which the brain deals with a dynamic world is to prepare multiple potential actions to available targets, presumably allowing each to be implemented in a moment s notice. If this is the case, however, then beyond merely encoding multiple movement directions, one would predict, given the importance of sensorimotor control policies to skilled motor behavior, that the brain may actually specify, in parallel, the feedback gains associated with each potential movement option. Indeed, according to OFC models 1,14,16, explicit motor planning involves setting higher level goals (for example, get the hand to the target) and specifying the parameters of the feedback controller before each movement. In addition, at the neural level, it is reasonable to assume that preparatory activity in motor areas, given direct spinal projections, encodes all aspects of the planned movement, including feedback gains associated with the control policy 48. We found that the visuomotor feedback gains implemented on lateselection trials (when each target still represents a potential reach option) were well approximated by the average of the gains for the narrow and wide targets on early-selection trials. This finding not only provides an extension of the spatial averaging phenomenon described above to the specification of sensorimotor control policies, but, more generally, provides a line of evidence in support for the parallel encoding framework suggested by the affordance competition hypothesis. Specifically, our results are consistent with the idea that the CNS prepares fully elaborated movements, complete with control policies governing feedback gains, for alternative reach options. Notably, the fact that an intermediate feedback gain was specified in late-selection trials suggests that participants were not overly concerned with maximizing the probability of hitting the target within the specified movement time (that is, based on automatic corrections), in which case they should have always set their gains for the narrow target. It is possible that implementing an intermediate gain will lower the cost of control associated with maintaining a high gain 49. Although spatial averaging behavior in go-before-you-know tasks is often taken as evidence that the brain encodes multiple potential movements in parallel 9, it has recently been suggested that it may instead constitute a deliberative strategy for minimizing movementrelated costs 27. That is, launching a single movement in the spatially averaged direction of potential targets tends to minimize, on average, the cost of in-flight corrective actions that must be taken when one of the potential targets is cued 29. It is difficult to imagine how people would similarly aim toward an average feedback gain, given that gains are concerned with the evolution of the movement and are not spatial in nature. It is conceivable that participants indirectly specify an intermediate gain by first constructing an average visual target (with an average width). However, we think this is unlikely, as there is strong evidence that, when required to launch a movement toward two potential targets in different spatial locations, participants do not construct a visually averaged target to aim toward 9. Furthermore, we found no evidence that intermediate gains result from learning (that is, average gains are seen from the earliest trials), which might be expected if participants developed a deliberate strategy of constructing an average visual target. Our results are consistent with a recently described model in which an optimal feedback control policy is calculated independently for each potential target and a weighted average of these policies (that is, feedback gains) is computed at each point in time based on the relative desirability of each target 5. Notably, this model, which predicts averaging of feedback gains, can also account for spatial (that is, trajectory) averaging in go-before-you-know tasks. We submit that our result showing feedback gain averaging, coupled with previous work demonstrating trajectory averaging, provides strong support for the compelling idea that the CNS, under cases of target uncertainty, encodes in parallel multiple motor plans, along with their associated control policies, for competing action options. nature NEUROSCIENCE VOLUME 19 NUMBER 2 FEBRUARY

7 npg 216 Nature America, Inc. All rights reserved. Methods Methods and any associated references are available in the online version of the paper. Note: Any Supplementary Information and Source Data files are available in the online version of the paper. Acknowledgments We thank D. Franklin for providing helpful advice and suggestions. This research was supported by the Natural Sciences and Engineering Research Council of Canada, the Wellcome Trust, the Human Frontiers Science Program, and by the Royal Society Noreen Murray Professorship in Neurobiology (to D.M.W.). J.P.G. was supported by a Canadian Institutes of Health Research fellowship award. AUTHORS CONTRIBUTIONS J.P.G., L.L., D.M.W. and J.R.F. designed the experiment. L.L. performed research. J.P.G., D.M.W. and J.R.F. analyzed and interpreted data. J.P.G., L.L., D.M.W. and J.R.F. wrote the paper. COMPETING FINANCIAL INTERESTS The authors declare no competing financial interests. Reprints and permissions information is available online at reprints/index.html. 1. Cisek, P. & Kalaska, J.F. Neural correlates of reaching decisions in dorsal premotor cortex: specification of multiple direction choices and final selection of action. Neuron 45, (25). 2. Thura, D. & Cisek, P. Deliberation and commitment in the premotor and primary motor cortex during dynamic decision making. Neuron 81, (214).. Cisek, P. Cortical mechanisms of action selection: the affordance competition hypothesis. Philos. Trans. R. Soc. Lond. B Biol. Sci. 62, (27). 4. Baumann, M.A., Fluet, M.C. & Scherberger, H. Context-specific grasp movement representation in the macaque anterior intraparietal area. J. Neurosci. 29, (29). 5. McPeek, R.M., Han, J.H. & Keller, E.L. Competition between saccade goals in the superior colliculus produces saccade curvature. J. Neurophysiol. 89, (2). 6. Chapman, C.S. et al. Reaching for the unknown: multiple target encoding and real-time decision-making in a rapid reach task. Cognition 116, (21). 7. Ghez, C. et al. Discrete and continuous planning of hand movements and isometric force trajectories. Exp. Brain Res. 115, (1997). 8. Van der Stigchel, S., Meeter, M. & Theeuwes, J. Eye movement trajectories and what they tell us. Neurosci. Biobehav. Rev., (26). 9. Stewart, B.M., Gallivan, J.P., Baugh, L.A. & Flanagan, J.R. Motor, not visual, encoding of potential reach targets. Curr. Biol. 24, R95 R954 (214). 1. Stewart, B.M., Baugh, L.A., Gallivan, J.P. & Flanagan, J.R. Simultaneous encoding of the direction and orientation of potential targets during reach planning: evidence of multiple competing reach plans. J. Neurophysiol. 11, (21). 11. Shadmehr, R., Smith, M.A. & Krakauer, J.W. Error correction, sensory prediction, and adaptation in motor control. Annu. Rev. Neurosci., (21). 12. Wolpert, D.M., Diedrichsen, J. & Flanagan, J.R. Principles of sensorimotor learning. Nat. Rev. Neurosci. 12, (211). 1. Scott, S.H. Optimal feedback control and the neural basis of volitional motor control. Nat. Rev. Neurosci. 5, (24). 14. Todorov, E. & Jordan, M.I. Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, (22). 15. Wolpert, D.M. & Flanagan, J.R. Motor learning. Curr. Biol. 2, R467 R472 (21). 16. Todorov, E. Optimality principles in sensorimotor control. Nat. Neurosci. 7, (24). 17. Cluff, T. & Scott, S.H. Rapid feedback responses correlate with reach adaptation and properties of novel upper limb loads. J. Neurosci., (21). 18. Franklin, S., Wolpert, D.M. & Franklin, D.W. Visuomotor feedback gains upregulate during the learning of novel dynamics. J. Neurophysiol. 18, (212). 19. Diedrichsen, J. Optimal task-dependent changes of bimanual feedback control and adaptation. Curr. Biol. 17, (27). 2. Dimitriou, M., Franklin, D.W. & Wolpert, D.M. Task-dependent coordination of rapid bimanual motor responses. J. Neurophysiol. 17, (212). 21. Pruszynski, J.A. et al. Primary motor cortex underlies multi-joint integration for fast feedback control. Nature 478, 87 9 (211). 22. Franklin, D.W. & Wolpert, D.M. Specificity of reflex adaptation for task-relevant variability. J. Neurosci. 28, (28). 2. Nashed, J.Y., Crevecoeur, F. & Scott, S.H. Influence of the behavioral goal and environmental obstacles on rapid feedback responses. J. Neurophysiol. 18, (212). 24. Knill, D.C., Bondada, A. & Chhabra, M. Flexible, task-dependent use of sensory feedback to control hand movements. J. Neurosci. 1, (211). 25. Dimitriou, M., Wolpert, D.M. & Franklin, D.W. The temporal evolution of feedback gains rapidly update to task demands. J. Neurosci., (21). 26. Smith, M.A., Ghazizadeh, A. & Shadmehr, R. Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 4, e179 (26). 27. Haith, A.M., Huberdeau, D.M. & Krakauer, J.W. Hedging your bets: intermediate movements as optimal behavior in the context of an incomplete decision. PLOS Comput. Biol. 11, e14171 (215). 28. Gallivan, J.P. et al. One to four, and nothing more: nonconscious parallel individuation of objects during action planning. Psychol. Sci. 22, (211). 29. Hudson, T.E., Maloney, L.T. & Landy, M.S. Movement planning with probabilistic target information. J. Neurophysiol. 98, 4 46 (27).. Izawa, J. & Shadmehr, R. On-line processing of uncertain information in visuomotor control. J. Neurosci. 28, (28). 1. Gallivan, J.P., Barton, K.S., Chapman, C.S., Wolpert, D.M. & Randall Flanagan, J. Action plan co-optimization reveals the parallel encoding of competing reach movements. Nat. Commun. 6, 7428 (215). 2. Corneil, B.D., Olivier, E. & Munoz, D.P. Visual responses on neck muscles reveal selective gating that prevents express saccades. Neuron 42, (24).. Saijo, N., Murakami, I., Nishida, S. & Gomi, H. Large-field visual motion directly induces an involuntary rapid manual following response. J. Neurosci. 25, (25). 4. Gomi, H., Abekawa, N. & Nishida, S. Spatiotemporal tuning of rapid interactions between visual-motion analysis and reaching movement. J. Neurosci. 26, (26). 5. Prablanc, C. & Martin, O. Automatic control during hand reaching at undetected two-dimensional target displacements. J. Neurophysiol. 67, (1992). 6. Day, B.L. & Lyon, I.N. Voluntary modification of automatic arm movements evoked by motion of a visual target. Exp. Brain Res. 1, (2). 7. Saunders, J.A. & Knill, D.C. Visual feedback control of hand movements. J. Neurosci. 24, (24). 8. Brenner, E. & Smeets, J.B. Perceptual requirements for fast manual responses. Exp. Brain Res. 15, (2). 9. Goodale, M.A., Pelisson, D. & Prablanc, C. Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature 2, (1986). 4. Körding, K.P. & Wolpert, D.M. Bayesian integration in sensorimotor learning. Nature 427, (24). 41. Braun, D.A., Aertsen, A., Wolpert, D.M. & Mehring, C. Learning optimal adaptation strategies in unpredictable motor tasks. J. Neurosci. 29, (29). 42. Orbán, G. & Wolpert, D.M. Representations of uncertainty in sensorimotor control. Curr. Opin. Neurobiol. 21, (211). 4. Franklin, D.W., Osu, R., Burdet, E., Kawato, M. & Milner, T.E. Adaptation to stable and unstable dynamics achieved by combined impedance control and inverse dynamics model. J. Neurophysiol. 9, (2). 44. Crevecoeur, F., McIntyre, J., Thonnard, J.L. & Lefèvre, P. Movement stability under uncertain internal models of dynamics. J. Neurophysiol. 14, (21). 45. Munoz, D.P. & Wurtz, R.H. Saccade-related activity in monkey superior colliculus. II. Spread of activity during saccades. J. Neurophysiol. 7, (1995). 46. Song, J.H. & Nakayama, K. Hidden cognitive states revealed in choice reaching tasks. Trends Cogn. Sci. 1, 6 66 (29). 47. Gallivan, J.P. & Chapman, C.S. Three-dimensional reach trajectories as a probe of real-time decision-making between multiple competing targets. Front. Neurosci. 8, 215 (214). 48. Churchland, M.M., Santhanam, G. & Shenoy, K.V. Preparatory activity in premotor and motor cortex reflects the speed of the upcoming reach. J. Neurophysiol. 96, (26). 49. Manohar, S.G. et al. Reward Pays the Cost of Noise Reduction in Motor and Cognitive Control. Curr. Biol. 25, (215). 5. Christopoulos, V. & Schrater, P.R. Dynamic Integration of Value Information into a Common Probability Currency as a Theory for Flexible Decision Making. PLOS Comput. Biol. 11, e1442 (215). 26 VOLUME 19 NUMBER 2 FEBRUARY 216 nature neuroscience

8 npg 216 Nature America, Inc. All rights reserved. ONLINE METHODS Participants. Eleven participants (five men and six women aged 18 2) participated in the experiment, with eight being included in data analysis (see participant exclusion criteria below). A target sample size of 1 12 participants was specified in advance based on previous studies in this area and our expectation that, if the main experimental effect was present, it should be observed in almost all participants. All participants were right-handed, had normal or corrected-to-normal vision, were neurologically healthy, and naive to the purpose of the study. They were compensated for their time with a cash payment of $2 2 (see below). The study was conducted with an experimental protocol approved by Queen s University Research Ethics Board, which adhered to the principles of the Canadian Tri-council Policy Statement on Ethical Conduct for Research Involving Humans and the principles of the Declaration of Helsinki (1964). Experimental apparatus. Participants were seated in a chair and used their right hand to grasp the handle of a robotic manipulandum (vbot 51 ) that could move freely in a horizontal plane (Fig. 1a). The robot measured the position of the handle and could apply forces to the hand via the handle. Participants were instructed to place their left hand comfortably in their lap. Targets and a cursor representing the position of the handle were displayed on a horizontal computer monitor located above the robot. The participant viewed these stimuli through a mirror located halfway between the monitor and handle, such that the stimuli appeared in the horizontal plane of the handle. The mirror prevented direct visual feedback of the participant s arm and the handle of the manipulandum. The position of the handle was calculated using joint position sensors on the vbot, which were sampled at 1 khz. Experimental design. Using the robotic handle to control the cursor, participants placed the cursor over a start location (both of which were represented as 1 cm diameter circles) positioned ~2 cm in front of their chest. Two potential, superimposed targets were located 25 cm directly in front of the start location: a narrow 2 cm 2 cm outlined square or a wide 8 cm 2 cm outlined rectangle (Fig. 1b). A cm 5 cm visual occluder (colored gray) was located midway between the start position and targets. On all trials, the cursor passed behind the occluder, emerging at the exact midpoint of the target distance (12.5 cm from the start position). On cursor jump trials, the cursor exited the occluder displaced by cm to the left or right (±x direction) of the hands position. We chose to perturb the cursor at the mid-point of target distance because it has been previously shown that participants reflex gains elicit the highest sensitivity at this point in the movement 25. Also, given that visuomotor gains are modulated as a function of distance to the target 25,52, the common perturbation point allowed for a direct comparison of the corrective responses across the different experimental conditions. Target selection. Our experiment consisted of both one- and two-target trials. In the one-target trials, only one of the two potential targets appeared at the beginning of each trial and appeared filled-in. In the two-target trials, both potential targets appeared at the beginning of the trial and were initially displayed as an outlined (unfilled) square and rectangle (superimposed) with a border thickness of mm. On early-selection trials, once the hand cursor reached 2.5 cm of target distance, one of the two targets filled in as the other simultaneously disappeared. On late-selection trials, the sequence of events was identical with the exception that the target was only filled in once the cursor had reached 22.5 cm of target distance. Thus, when the cursor was perturbed (at 12.5-cm distance to target) on these late-selection trials, participants could correct for the cursor perturbation before actually knowing which target would be eventually cued. Critically, on both early and late-selection trials, each target had an equal likelihood of being cued. Force channel and non-channel trials. On the majority of trials, the motion of the vbot handle, and thus the cursor on the screen, was entirely controlled by the participant. Thus, on trials in which the cursor was perturbed, in order for the participant to reach the target, they needed to implement corrective responses by moving the handle (Fig. 1b). To clearly assess participants reflex gains, we also, on a minority of trials, incorporated a force channel, which has been used in previous research to examine corrective actions uncontaminated by limb dynamics (for example, see refs. 22,25,26). The force channel was used to mechanically constrain the handle (and thus the participant s hand) to the direct path between the start location and the target. The two walls of the channel were simulated as stiff damped springs (with a stiffness of 4, N m 1 and a viscosity of 8 N m 1 s 1 ) that prevented the handle from moving laterally (Fig. 1c). The force applied by the robot in order to keep the hand mechanically constrained in the channel, which is equal and opposite the lateral force that the participant applies to the wall of the channel in response to a cursor perturbation, provides a direct read-out of the visuomotor feedback gain that is uncontaminated by lateral motion of the hand. In force channel trials with a cursor perturbation (Fig. 1c), the cursor remained offset (after emerging from the occluder) for 25 ms and then automatically returned to its midline position, allowing participants to successfully complete the task. The time of 25 ms was chosen on the basis of previous work 25, and allowed us an adequate time window in which to measure the automatic reflex response to the perturbation. Whereas the vast majority of participants, at postexperiment debriefing, seemed to have no explicit knowledge that the cursor correction (at 25 ms after perturbation) was not driven by their own movement, all participants appeared to be unaware that their hand movement path had in fact been constrained (to a straight-ahead movement) on that subset of trials. This is consistent with previous work that has used channel trials to explore feedback gain modulation 22,25 and force-field adaptation 26, and may also reflect the fact that our force channel cursor perturbation trials occurred relatively infrequently during testing (for details, see below). Experimental timing. Once the cursor was positioned over the start location for 2 ms, a progression of five beeps, each spaced 6 ms apart, were played. Participants were instructed to leave the start position on the fourth beep and arrive at the target on the fifth beep. If the reach was initiated too quickly (>2 ms before the fourth beep) or too slowly (>2 ms after the fourth beep), participants, upon completion of the trial, were given the text feedback Too Early or Too Late, respectively, on the screen. If the total movement duration (time from start position to target) was greater than 8 ms, participants were given the text feedback Too Slow following the trial. If the total movement duration was less than 4 ms, participants were given the text feedback Too Fast following the trial. On trials in which participants did not commit these timing errors, and depending on whether participants hit or missed the target (the criterion for a hit being whether the pixels of the cursor overlapped with the pixels of the target), the text feedback Good or Miss, respectively, was displayed following the trial. This feedback, in addition to encouraging similar accuracy demands across participants, encouraged consistent timing across trials. For instance, the amount of time it took (in channel trials) for participants to move, in the direction of the target, from to 25 mm (that is, from the start position to the point where the early target selection occurred), from 25 to 125 mm (the point where the perturbation occurred), from 125 to 24 mm (the front edge of the target) and from 225 to 24 mm (from the point where the late target selection occurred to the front edge of the target) were as follows (range of the lowest-to-highest and average movement duration, based on participant medians): to 25 mm: ms, M = 19 ms; 25 to 125 mm: ms, M = 19 ms; 125 to 24 mm: ms, M = 269 ms; 225 to 24 mm: ms, M = 6 ms). When adding each of the first three mean times together, we obtained a 598-ms movement time, only 2 ms less than 6 ms, the time interval between the fourth and fifth beeps (the first providing the cue for participants to move and the latter providing the desired target contact time). Given the finding from a previous study 25 that individuals can reliably update their reflex gains within 1 ms, this consistency in the experimental timing of our task is important in several ways. First, it allowed participants, on early-selection trials, adequate time (mean of 19 ms) between target cuing (at 25 mm of reach distance) and cursor perturbation (at 125 mm of reach distance) to reliably update their visuomotor feedback gains in accordance with the cued target size. Second, given that the average duration between the time of cursor perturbation and late target selection is approximately 21 ms (that is, 269 ms minus 6 ms), it is highly unlikely that the visual perturbation-induced corrective responses measured on these two-target trials (at 18 2 ms post perturbation onset) is contaminated by any of the subsequent (voluntary) corrective responses associated with target cuing that occurs later on during the movement (which would require, at minimum, an additional ~1 ms to be implemented, that is, 1 ms). Third, the time between target cuing and target contact on the late-selection trials (mean of 6 ms) also makes it highly unlikely that participants could have deliberately doi:1.18/nn.4214 nature NEUROSCIENCE