Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 6 Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model JEFFREY S. JOHNSON AND VANESSA R. SIMMERING hallmark of perceptual processes is that they A remain tightly linked to the world, allowing adaptive changes in behavior in the face of changing environmental circumstances for example, in the detection and exploration of visible objects, the continuous tracking of moving objects, and so forth. In contrast, a defining feature of higher-level cognitive processes such as working memory is a degree of decoupling from input such that behavior relies on the internal state of the organism (reflecting, for instance, behavioral goals), rather than being continuously locked onto incoming information from the environment. If cognitive processes are to serve adaptive behavior, however, they must be integrated with perceptual processes in a coordinated fashion. For example, when deciding when to turn while driving, it is not enough to follow a planned and/or remembered route; you must also judge the conditions of traffic as they change and adjust your actions to control the vehicle appropriately under those conditions, incorporating your action plan with the current perceptual input. In this chapter, we consider these challenges within the context of two behavioral paradigms that provide an ideal setting for probing the integration of perception and working memory: spatial recall and change detection. In a typical spatial recall task (Figure 6.1a), participants are asked to remember the location of a single object (e.g., a small black dot) brief ly presented on a computer monitor or tabletop. Following a short delay interval (e.g., 5 2 seconds), they are asked to report the remembered location using, for instance, a computer mouse. In the change detection task (Figure 6.1b), participants view brief ly presented memory arrays consisting of one or more simple objects (e.g., colored squares). After a short memory delay, a test array is presented, and the participant must compare the test array with the memory array to identify whether the arrays are the same or different. For the former task, the primary challenge lies in keeping your memory of the object s location grounded in the relevant spatial reference frame. For the latter, the challenge lies in comparing what you are seeing now with what you saw before, and detecting relevant changes when they occur. Although these challenges sound different on the surface, they share two common features. First, they both require maintaining a particular stable state representing relevant features of the environment when those features are no longer perceptually available and ignoring other (potentially distracting) environmental events. Second, both tasks require interfacing with the real world in a coordinated fashion by locking onto reference frames in the case of spatial tasks, and by making appropriate comparisons between items in memory and current perceptual information in the case of change detection. In the sections that follow, we describe a dynamic field architecture that addresses the integration of perception and working memory, and we illustrate how this architecture functions in the context of spatial recall and change detection tasks. Within these sections, we describe how this framework has been (or can be) used to address key aspects of each task, including the encoding and maintenance of information in working memory, the comparison of working memory representations with perceptual inputs, and the generation of response-related decisions. We conclude with ideas and directions for future research building on the dynamic processes embodied in the three-layer architecture. 2_med_9781993563_part_2.indd 151 8/3/215 5:25:5 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 152 Integrating Lower-Level Perception-Action (a) Memory array Delay Recall (.5 2 s) (5 2 s) (untill response) (b) Memory array Delay Test Array (1 5 ms) (25 9 ms) (untill response) Same or different? FIGURE 6.1: The spatial recall (a) and change detection (b) tasks used to study working memory for spatial locations and object features, respectively. I N T E G R AT I N G P E R C E P T U A L A N D M E M O R Y P R O C E S S E S IN A T H R E E - L AY E R N E U R A L FIELD ARCHITECTURE Schutte and Schöner (Chapter 2, this volume; see also Amari, 1977; Amari & Arbib, 1977) described a simple two-layer dynamic field model that can serve either a perceptual or a working memory function, depending on the specific dynamic mode the model is working within. When functioning in a self-stabilized, or input-driven, mode, peaks of activation representing, for instance, the color or location of a perceived object can be formed and maintained as long as input is present. However, once input is removed, activation within the field quickly transitions back to a stable baseline resting state. This mode of functioning is ideal for capturing elementary perceptual processes, such as detecting and identifying a particular object in the task space, tracking its movement, and so forth. With fairly subtle changes to the network dynamics (e.g., a small change in the strength of local excitation), however, the model may also function in a self-sustained, or memory-driven, mode, in which peaks of activation can remain above threshold in the absence of continuing input. This mode of functioning is central to capturing phenomena related to working memory in DFT. Thus, depending on its mode of operation, the two-layer network can serve either a perceptual or a working memory function, but not both. To capture performance in spatial recall and change detection tasks, however, it is necessary to integrate both functions in a single architecture. To address this challenge, we have developed the three-layer dynamic field model depicted in Figure 6.2 (see Box 6.1 for a formal description; for additional discussion, see Johnson, Simmering, & Buss, 214; Johnson, Spencer, & Schöner, 28, 29; Simmering, Schutte, & Spencer, 28; Spencer, Simmering, Schutte, & Schöner, 27). The basic model consists of an excitatory contrast field (CON(u); Figure 6.2b), an excitatory working memory field (WM(w); Figure 6.2d), and a shared inhibitory field (Inhib(v); Figure 6.2c). In each field, the x-axis is spanned by a collection of activation variables defined over particular metric feature dimensions (e.g., color, location, direction of motion), the y-axis shows the activation level of each activation variable, and the z-axis depicts the elapsed time since the beginning of the simulated trial. These layers pass excitation and inhibition as indicated by solid and dashed arrows, respectively. CON is the primary target of perceptual input to the model, although the WM field also receives weak direct input. Additionally, as discussed in previous chapters, neighboring sites in both fields (i.e., sites coding for similar properties) interact via local excitatory connections. With respect to coupling among the layers, CON provides the primary source of excitatory input to both Inhib and WM, and Inhib provides inhibitory input to both CON and WM. Critically, WM only passes activation to CON via the inhibitory layer. That is, the only external source of excitatory input to CON consists of direct stimulus input. Thus, CON is primarily excited by direct afferent input, whereas WM is primarily excited by input from CON (with weak direct input). These differences lead to the emergence of different functional roles, with CON playing a primarily perceptual role (e.g., detecting new inputs, contrasting 2_med_9781993563_part_2.indd 152 8/3/215 5:25:5 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 153 (a) 5 4 3 2 Target Input 1 Midline 1 18 9 9 18 (b) 2 2 18 9 (c) Midline peak CON 9 18 Inhib 12 8 4 5 5 18 9 9 18 WM 12 8 4 Time [s] (d) 2 2 Location [ ] Drift away from midline 18 9 9 18 12 8 4 FIGURE 6.2: Simulation of a spatial recall trial. (a) Inputs corresponding to the midline and target are presented to the three-layer model (b d). Input from midline forms a peak in the contrast field (CON) (b) throughout the trial, and corresponding inhibition in the inhibitory field (Inhib) (c) projects to both CON and the working memory field (WM) (c). While the peak corresponding to the target input is sustained during the delay, inhibition associated with midline repels the peak, leading to an error in which memory is biased away from midline. Excitatory and inhibitory interactions among the model s layers are indicated by solid and dashed arrows, respectively. new inputs with memorized values), and WM serving to maintain a memory of previously encoded stimuli. Taken together, therefore, the three-layer architecture provides each of the components needed to capture performance in spatial recall and change detection tasks. In the following sections, we walk through simulations of the model performing individual trials of each task. We also highlight how the framework has been used to capture key phenomena in the literature and to generate novel predictions that have been tested in behavioral experiments. SPATIAL RECALL BIASES Spatial recall tasks were one of the first applications of the three-layer architecture (Spencer et al., 27). In a typical task, a target is presented within a homogeneous task space, and participants remember the location for a short (5- to 2-second) delay before responding by pointing or using a computer mouse to move a cursor to the remembered location. In these tasks, adults show delay-dependent biases such that memories for location drift systematically away from visible edges and symmetry axes as the delay is increased (e.g., Engebretson & 2_med_9781993563_part_2.indd 153 8/3/215 5:25:5 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 154 Integrating Lower-Level Perception-Action BOX 6.1 DF THREE-LAYER ARCHITECTURE The three-layer model consists of an excitatory layer, u(x; t), which receives afferent sensory input, S(x; t); a shared inhibitory layer, v(x; t); and second excitatory layer, w(x; t) that receives excitatory input, primarily from the first excitatory layer but also a weak copy of the sensory input. τ u( xt, ) = u( xt, ) + h + S( xt, ) + dx k ( x x ) g( ux (, t) ) dxk cuv dxk uv( x x ) gvx ( (, t) )+ kr ( x x ) ξ( x, t) dx global ( x x ) gvx ( (, t) ) u uu uv τ v( xt, ) = v( xt, ) + hv + dxk vu( x x ) gux ( (, t) )+ dx kvw ( x x ) g ( ux (, t) ) ( ) ξ (, ) + k x x x tdx r The kernels, k x x, projecting across the levels i j ij ( ) ( ) or mediating interactions within excitatory layers (i = j) are all Gaussian with positive strength, but may vary in width and ( ) x x strength: cij ( x x ) = cijstrengthexp 2 2σij 2,,. External input localized around x input is supplied ( ) x x 2 only to the u layer in the form: Sxt (, ) = astrengthexp. The sigmoidal function is given by 2 2ó input 1 gu ( ) =. 1 + exp β u [ u ] Huttenlocher, 1996; Lipinski, Simmering, Johnson, & Spencer, 21; Spencer & Hund, 22; Tversky & Schiano, 1989; Werner & Diedrichsen, 22). In addition, memory for locations aligned with a reference axis such as a target presented on the midline symmetry axis of a task space is stable, showing low mean error and low variability across repeated trials (e.g., Spencer & Hund, 22). Performing a spatial recall trial requires encoding the location relative to a local reference frame (i.e., edges and symmetry axes of the computer monitor in Figure 6.1a), maintaining the location in memory over the delay, and then reproducing the location within the reference frame. Thus, coordination between the memory of the location and the reference frame is critical to performing the task. To illustrate how the three-layer DF model achieves this coordination, Figure 6.2 shows a simulation of the DF model performing a single spatial recall trial (Simmering et al., 28). Before the target appears, the edges and midline symmetry axis are inputs to CON; note that, for simplicity, we show only the midline input (see midline at in Figure 6.2a). This input projects strongly into CON and weakly into WM, and drives the formation of a self-stabilized peak in CON (Figure 6.2b). The peak in CON projects activation to the same spatial position in Inhib (Figure 6.2c); activation from Inhib then projects broad inhibition to both CON and WM. The trial proceeds with presentation of the target for 2 seconds (see target in Figure 6.2a), which projects strong input to the model (strongly into CON, weakly into WM; see arrows from Figures 6.2a to 6.2b and to 6.2d) at 35 in the task space. The target input forms peaks in both CON and WM (Figure 6.2d), which sends activation to Inhib. As activation builds in Inhib, inhibition projects back to both CON and WM. When the target input is removed (i.e., the target disappears from the task space), the peak in CON drops below threshold while the peak in WM remains in a self-sustaining state due to the different strength of neural interactions between these layers. As this peak is maintained in WM over the 1-second delay, the inhibition associated with the midline peak in CON (Figure 6.2b) creates stronger inhibition in WM around midline than elsewhere in this field. Thus, local excitatory interactions are stronger on the non-midline side of the WM peak, while inhibitory interactions are stronger on the midline side of the peak. This effectively repels the WM 2_med_9781993563_part_2.indd 154 8/3/215 5:25:5 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 155 peak away from the midline peak in CON over the course of the delay (see drift arrow in Figure 6.2d). As a result, at the end of the trial, the peak has drifted to 46 ; the model responds by reading out the location of this peak, resulting in an 11 error away from midline. The real-time processes in the DF model illustrate how the bias away from midline emerges over the course of the delay. As Figure 6.2d shows, the WM peak drifts away from the midline peak in CON continuously; if the model s memory were probed at 5 seconds rather than 1 seconds, for example, the magnitude of drift would be smaller. Importantly, however, this drift has spatial limitations both near and far from midline. Specifically, when targets are aligned with midline (i.e., at ), the excitatory component of the midline peak in CON is strong enough to stabilize the peak in WM, counteracting the effect of inhibition and resulting in little to no drift. On the other hand, if targets are presented far from midline (i.e., 6 8 ), the spread of inhibition around midline does not reach far enough to significantly influence the maintenance of the WM peak. Rather, as such WM peaks sustain, noise causes random fluctuation in the position of the peak, resulting in unsystematic errors around the true location (see Spencer & Hund, 22, for behavioral results consistent with these patterns). Stochastic processes at work in the model also give rise to variation in the magnitude of drift across trials. For example, if the trial in Figure 6.2 is presented repeatedly to the model, drift may vary from 3 to 15 on any given trial. This would result in high variable error, that is, the standard deviation across repeated trials within a single individual participant. In spatial regions where excitation is stronger relative to inhibition (i.e., near or at midline), the variability of responses across trials is typically lower (Spencer & Hund, 22). When both inhibition and excitation associated with midline are weak (i.e., far from midline), mean errors are small because drift is unsystematic, but variable error is still high (Spencer & Hund, 22). Thus, the three-layer architecture captures a complex pattern of drift and variability across locations in spatial recall tasks. Before we developed the three-layer model, we considered alternative DF models, but these other models failed to capture the full range of performance in spatial recall. For example, one previous version of the DF model (described by Simmering, Spencer, & Schöner, 26), achieved repulsion from midline through an inhibitory memory trace (similar to the excitatory memory trace described in Chapter 2). Behavioral results from Simmering and Spencer (27), however, showed that repulsion in memory arises on a trial-by-trial basis depending on the perceptual structure of the task space. If, for instance, a reference axis was supported by presenting two dots in otherwise empty space, the effect on recall was similar to that of the midline symmetry axis; on an immediately subsequent trial, if these dots were removed, performance showed no influence of the previously available reference axis. Thus, repulsion is driven by the current perceptual structure, rather than by the slower process of building an inhibitory memory trace. These behavioral results ruled out one type of DF model in favor of the three-layer model described here. This highlights how useful it can be to have a tight link between a theoretical model and empirical work the data in this case provided key constraints that favored one model architecture over another. This also highlights that developing theories is an ongoing process that requires continual empirical testing and model refinement as new findings are revealed. VISUAL CHANGE DETECTION Like the spatial recall task considered in the last section, successful performance of the change detection task (see Figure 6.1b) requires the encoding of memory array items into WM and their maintenance throughout the delay interval. In addition to these processes, change detection also requires the comparison of items in memory with the perceptual information available in the test array, and the generation of a same or different decision once the test array is presented. The simulations shown in Figure 6.3 illustrate how each of these processes arises in the three-layer model. For clarity, each simulation shows the state of activation (y-axis) across each activation variable (x-axis) at important time points in the trial: following encoding (Figure 6.3a), during maintenance of information over the delay interval (Figure 6.3b), and during generation of a response at testing (Figure 6.3c d). Additionally, the lower panels show the state of activation of individual nodes in a response layer that was added to the model to capture the same or different decision required by the task (Figure 6.3e f). At the beginning of the trial, three inputs representing the appearance of three colored squares in the memory array are provided to the model (dashed curves in Figure 6.3a). This event pushes activation above threshold at three field locations 2_med_9781993563_part_2.indd 155 8/3/215 5:25:51 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 156 Integrating Lower-Level Perception-Action (a) 4 2 CON 2 4 18 9 9 18 4 2 Inhib 2 4 18 9 9 18 4 2 WM 2 4 18 9 9 18 4 (b) 2 CON 2 4 18 9 9 18 4 2 Inhib 2 4 18 9 9 18 4 2 WM 2 4 18 9 9 18 Color [ ] Color [ ] (c) 4 2 CON 2 4 18 9 9 18 4 2 Inhib 2 4 18 9 9 18 4 2 WM 2 4 18 9 9 18 (d) 4 2 CON 2 4 18 9 9 18 4 2 Inhib 2 4 18 9 9 18 4 2 WM 2 4 18 9 9 18 Color [ ] Color [ ] (e) 4 1 2 2 1 2 3 4 5 1 2 3 4 5 Time [s] Same Different 4 1 Time [s] FIGURE 6.3: Simulation of two change detection trials. Time slices through each layer show critical points in the trials: (a) encoding three targets, at the end of the memory array presentation; (b) maintaining the colors across the delay interval; and comparison of the contents of WM with three inputs corresponding to (c) a no-change test array and (d) a change test array. Also shown is activation of the decision nodes following the (e) no-change and (f) change test arrays. Arrows indicate progression through the trial(s). Dashed lines in each panel indicate the activation threshold (i.e., ). Abbreviations as in Figure 6.2. (f) Different Same in CON, representing the model s detection of the features in the task space. Once above-threshold activation is present in CON, strong activation projects to both Inhib and WM. The interplay of excitation and inhibition between CON and Inhib momentarily stabilizes the peaks of activation in CON. However, once activation exceeds threshold in WM, the extra boost of excitation to Inhib begins to suppress the peaks in CON. As a result, when the input to the model is removed (Figure 6.3a), the peaks of activation in CON are suppressed below baseline, whereas three above-threshold peaks remain in WM. Locally excitatory interactions among field sites within WM together with lateral inhibition from Inhib allow these peaks to be sustained throughout the delay interval (Figure 6.3b). By contrast, strong inhibitory feedback from Inhib to CON produces localized troughs of inhibition in CON centered at the values being maintained in WM. 2_med_9781993563_part_2.indd 156 8/3/215 5:25:51 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 157 Next, the three colors within the test array are presented to the model. On no-change trials, these three colors are identical to those presented in the memory array (see dashed curves in Figure 6.3c). Because the model is holding these three colors in WM, when they are presented at testing, the inhibitory troughs in CON prevent activation from piercing threshold (Figure 6.3c). By contrast, if one item changes to a new color as on change trials (see dashed curves in Figure 6.3d), this input comes into CON at a relatively uninhibited field site. As a result, an above-threshold peak associated with the new color is able to form in CON at testing (see circle in Figure 6.3d). Localized inhibitory feedback to CON during the delay allows the three-layer model to function as a change detector, only building new peaks of activation in CON when the test array contains features that are not currently in WM. To generate the same or different response required by the task, however, the differential state of activation in CON versus WM at test must be translated into a binary decision. This is achieved by coupling the excitatory layers of the three-layer model to a simple competitive neural accumulator model (see, e.g., Standage, You, Wang, & Dorris, 211; Usher & McClelland, 21), comprised of two self-excitatory and mutually inhibitory nodes that act like a discretized form of the dynamic field (see Chapter 1). One node receives summed excitatory activation from CON to generate a different response, whereas the other receives summed activation from WM to generate a same response. Connections between the excitatory layers and the response layer are autonomously enabled when the test array input is turned on, and competition between the nodes ensures that only one becomes active at testing. Thus, the model s response ref lects the outcome of competitive interactions between activation projected from CON, which detects changing items in the perceptual array, as well as WM, which represents the current contents of memory. In the simulations shown in Figure 6.3, when the test input matches the contents of memory (Figure 6.3c), the primary input to the response layer comes from WM. This allows activation of the same node to exceed threshold at testing (Figure 6.3e), giving rise to a correct same response. By contrast, when a new item forms a peak in CON, the response layer receives inputs from both excitatory layers. The connection between CON and the different node is tuned to be stronger than the connection from WM to the same node, and, as a result, the different node wins the competition for response output, generating a correct different response (Figure 6.3f). B E H AV I O R A L S I G N AT U R E S O F E N C O D I N G, M A I N T E N A N C E, AND COMPARISON IN T H E THREE-LAYER MODEL The simulations just described show how the three-layer model captures each of the processes required by recall and change detection tasks used to study working memory. In this section, we describe how the model has been (or can be) used to address behavioral phenomena related to the encoding, maintenance, comparison, and decision processes implemented in the model. Encoding The DF model described here predicts that performance in laboratory tasks like change detection and recall will vary as a function of specific experimental conditions, such as the number of items that need to be remembered, their metric similarity, and timing. To probe these task-specific properties, the model is provided with a series of inputs, and the state of the model at the end of each simulated trial is recorded and used to calculate the model s performance. Because activation within the model evolves continuously under the influence of task input and within- and between-field interactions, it is possible to systematically evaluate the impact of specific experimental manipulations on different aspects of each task. For example, one well-established finding is that the rate of encoding information in WM increases as a function of the number of items to be encoded, referred to as the set size (Vogel, Woodman, & Luck, 26). How might this be accounted for in the DF model? Recall that successful WM encoding in the three-layer architecture can be said to have occurred when a stable peak of activation forms in the WM layer. This occurs as activation at stimulated field sites gradually increases above baseline. Assuming input is sufficiently strong and prolonged, the field will then transition from the stable baseline resting state to the self-sustained state, in which peaks of activation are able to remain above threshold in the absence of continuing external input. The sudden transition from the baseline state to the self-sustained state arises as a result of the nonlinear sigmoidal function governing the 2_med_9781993563_part_2.indd 157 8/3/215 5:25:51 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 158 Integrating Lower-Level Perception-Action 18 16 14 12 1 8 6 SS1 SS2 SS3 SS4 SS5 SS6 4 2 t5 t61 t72 t83 t94 t15 t116 t127 t138 t149 t16 t171 t182 t193 t24 t215 t226 t237 t248 t259 t27 t281 t292 Simulation timestep FIGURE 6.4: Rise time of activation in the working memory layer of the three-layer model as a function of the number of inputs presented to the model (i.e., set size [SS]). builds more slowly with greater numbers of inputs due to increased lateral inhibitory interactions among peaks. This property can account for the observation of slowed consolidation rates as a function of set size in laboratory experiments of change detection (Vogel et al., 26). Gray shading depicts the standard error of the mean (for clarity, only positive values are shown). transmission of activation within and between fields. Specifically, activation levels in each model layer are gated such that only sites with positive levels of activation transmit their state to other neural neighbors or layers. As a result, while activation in the field remains negative, activation at stimulated sites increases gradually under the inf luence of stimulus input. Once activation reaches threshold, however, the evolution of activation within the field is strongly inf luenced by excitatory and inhibitory interactions within and between layers. Specifically, strong locally excitatory interactions among field sites within the excitatory layers of the model produce a sudden dramatic increase in activation at stimulated field sites, while slower acting inhibitory feedback prevents the diffusion of activation throughout the field. As more items are presented to the model and more peaks are built, however, lateral inhibition begins to spread more broadly and becomes stronger, which can slow down or, in some cases, prevent the growth of peaks. Figure 6.4 shows the rise time of activation within the WM layer of the model as a function of the number of inputs to the model. As can be seen, activation builds more slowly as set size increases, in keeping with behavioral observations (Vogel et al., 26). Other factors that may be expected to inf luence the rate of encoding may include the metric relationship between the items in the memory array, the relative salience of particular items in the memory array, and the state of attention at the time of encoding. Maintenance The three-layer model has also been used to account for various factors related to the maintenance of information in WM. For instance, as discussed in the section on spatial recall, the basic stabilization mechanism at work in the model (i.e., local excitation and lateral inhibition) can produce drift of WM peaks when other inputs are present in the field (e.g., a midline reference input). Building on this work, Simmering and colleagues (26) tested several predictions derived from a DF model of position discrimination that was a precursor to the three-layer model described here. 1 Specifically, they studied the effect of reference axes on peak shape and drift and their impact on spatial discrimination performance. To assess spatial discrimination, participants were presented with two dots in quick succession and were asked to report whether the dots appeared in the same or different locations. Across a series of experiments, the position of the two dots relative to available reference cues was manipulated, as was the relative strength and position of reference cues. The strength of the reference axis was manipulated by leaving it unmarked, by 1 Note that the model of position discrimination depicted in Figure 6.5 and described in the corresponding text (described further in Simmering et al., 28; Simmering & Spencer, 28) represents a modification of the original model described in Simmering et al. (26). It has been updated to conform to the three-layer model of change detection described in this chapter. 2_med_9781993563_part_2.indd 158 8/3/215 5:25:51 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 159 marking it with two dots at the bottom and top of the display area, or by marking it with a dashed line at the top or bottom of the screen (see Simmering et al., 26). Model simulations of this task generated three predictions. First, spatial discrimination should be enhanced (i.e., discrimination thresholds are lower) when stimuli are presented toward versus away from the reference axis, due to the influence of drift (illustrated in Figure 6.5). Second, discrimination should be enhanced for stimuli presented near versus far from the reference axis, due to reference-related inhibition. Third, this enhancement effect should increase as the salience of the reference axis was increased, and similar effects should be observed when the reference inputs are presented around a lateral axis, shifted 15 cm to the right of midline. How do these effects arise in the model? Figure 6.5 illustrates the model s performance in this task. As in the previous simulation examples, presentation of the first stimulus input forms a self-sustaining peak in WM (Figure 6.5B) and produces a corresponding trough of inhibition in CON (Figure 6.5A). In contrast to the change detection simulations, however, the spatial discrimination task also includes input from the reference axis (e.g., midline). As in the spatial recall task described earlier, this constant input produces inhibition that repels the WM peak during the memory delay. Although the delay in the discrimination task is shorter (5 ms), a small amount of drift still occurs. Thus, when the second stimulus input is presented, the WM peak has drifted away from the reference axis. If the second stimulus is presented in the direction of drift (that is, away from the reference axis), the input will overlap with the position of the peak in WM (Figure 6.5D), preventing activation from piercing threshold in CON (Figure 6.5C). As in the change detection task, this leads the model to respond same. By contrast, if the second stimulus is presented toward the reference axis, in the opposite direction of drift, the input will not overlap with the peak in WM and instead will form a new peak in CON (Figure 6.5E) and the model will respond different. Importantly, this contrast in responses based on the direction in which the second stimulus FIGURE 6.5: Simulation illustrating the inf luence of drift on spatial discrimination. Adapted from Simmering and Spencer (28). 2_med_9781993563_part_2.indd 159 8/3/215 5:25:52 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 16 Integrating Lower-Level Perception-Action is presented occurs for identical spatial separations between the first and second stimuli. This illustrates the first prediction of the model, that discrimination is enhanced when stimuli are presented toward the reference axis (Figure 6.5G). The second and third predictions of the model also arise through reference-related inhibition. When stimuli are presented near the reference axis, inhibition is stronger and is symmetrical around the memory peak. In this case, rather than producing drift away from the reference, the memory peak is sharpened and stabilized by the inhibition, which makes it easier to detect relatively small displacements of target position (see behavioral prediction in Figure 6.5H). Importantly, because both peak width and the direction of drift arise through reference-related inhibition in the model, they should combine to inf luence behavior (Figure 6.5I). Moreover, as the strength of the reference is increased by adding perceptual structure (e.g., two dots aligned with midline), the sharpening of the memory peak also increases, producing graded improvements in position discrimination. These effects of reference-related inhibition are not unique to midline, but can be induced by providing perceptual structure in otherwise empty space. A series of behavioral studies confirmed all three predictions of the model (Simmering et al., 26). In addition to being influenced by perceived reference frames, interactions among peaks can give rise to distortions or enhancements of memory representations when multiple items are present in WM simultaneously. For instance, when two very similar inputs are presented to the model, the locally excitatory regions of each peak will overlap, causing them to fuse into a single peak. At more intermediate separations, field sites in between each peak will be inhibited to a greater extent than field sites on the outside edge of each peak. As a result, the peaks will tend to be repelled away from each other over the delay (see Figure 6.6a). This leads to the prediction that, when similar items (e.g., colors) are held simultaneously in WM, they will be recalled as being more distinct than they actually are. Finally, when peaks are far apart from one another, recall responses will be unbiased. To test these predictions of the model, Johnson and colleagues (Johnson et al., 28; Johnson, Dineva, & Spencer, 214), used the color recall paradigm depicted in Figure 6.6b. In this task, participants were presented with a sample array that contained three colored targets one unique and two similar and were cued to recall one of them on each trial by selecting the remembered color from a continuous color wheel using a computer mouse. When one of the similar color targets was cued at testing, recall responses were strongly biased in a direction away from the other similar color value being held in memory. By contrast, errors to the unique target showed a small negative bias. Thus, empirical data confirmed the model predictions. Comparison The modeling framework described here has also been used to understand the inf luence of metric interactions among items in WM on the comparison process (Johnson, Spencer, Luck, & Schöner, 29). Recall that change detection decisions rely critically on interactions between CON and WM via the inhibitory layer. In turn, the position, strength, and width of the inhibitory projection back to CON depends on what is being held in WM. As discussed in the last section, items in WM are not stored independently, but interact in specific ways depending on their metric similarity. In particular, we considered the case where inhibitory interactions between two nearby colors produced systematic distortions of each memory representation. Another consequence of laterally inhibitory interactions among items is an overall reduction in amplitude and a sharpening of the memory representation of each item (compare the amplitude and width of the two similar peaks to the unique peak in Figure 6.6a). In the context of change detection, this results in a narrower and shallower projection from WM to CON via Inhib (compare inhibitory troughs in CON in Figure 6.7b). As a consequence, it is easier to form a peak in CON when one of the similar items is changed to a new color at testing (compare Figures 6.7d and 6.7f). Thus, the model makes the counterintuitive prediction that change detection performance should be enhanced (i.e., accuracy should be higher) when multiple highly similar items, versus unique items, are stored. This prediction was confirmed in two separate experiments probing memory for colors and orientations. Finally, the three-layer model has also been used to address the question of capacity limits in WM, and the related question of how errors arise in the change detection task (Johnson, Simmering, & Buss, 214). A pervasive finding in the literature is that performance in the change detection task becomes worse as the amount of information that needs to be remembered is increased (see review 2_med_9781993563_part_2.indd 16 8/3/215 5:25:52 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 161 (a) Max Mutual repulsion No systematic error Stronger inhibition between peaks Min 18 Color [ ] 36 End of sample End of delay (b) Memory array Delay Test 8 ms 1 ms Until response FIGURE 6.6: Simulation showing mutual repulsion between nearby peaks in the working memory layer of the three-layer model. Overlapping inhibition between nearby peaks produces stronger inhibition in between the peaks than on either side, allowing activation to grow more easily in the direction away from the other peak (a). As a consequence, peaks will tend to move away from each other (i.e., they will repel each other) across the delay interval (compare solid versus dashed lines in panel a). This prediction of the model was tested using a color recall task (b) in which participants remembered three colors (two similar and one distinct) across a short delay interval, followed by the appearance of a color wheel cuing them to recall the color that had appeared at that location in the memory array. Results revealed that recall responses for the nearby colors were biased in opposite directions, whereas estimates of the unique color were not biased, in keeping with the model s predictions. in Cowan, 25). This finding has been taken as support for the proposal that WM is capacity limited. This property of WM can be accounted for by the DFT as a result of increasing inhibition as the number of inputs to the model increases. Beyond a certain point, inhibition begins to overwhelm excitation, and as a result, one or more peaks of activation either fail to build or fail to sustain throughout the delay interval. This failure to encode or maintain peaks throughout the delay may give rise to errors at testing. For instance, when a test input matches a forgotten item, a peak may build in CON, producing an incorrect different response. Similarly, when WM is full to capacity, inhibition spreads broadly throughout CON, which can make it difficult to build above-threshold peaks at testing, even when the test input does not match the contents of WM (i.e., when a change has occurred). Thus, in addition to accounting for capacity limits through the number of items that can be encoded and maintained, the model makes specific predictions regarding the source of particular types of errors in the change detection task that may arise through the comparison process as well. These more general issues can be explored by varying the number of items presented to the model, their metric separation, and exposure duration (see Exercise 3 at the end of this chapter). Additionally, the impact of changes in the balance of excitation and inhibition on capacity and performance of the change detection task can be explored by comparing the performance of the model with different parameter settings, as recommended in Exercise 1. 2_med_9781993563_part_2.indd 161 8/3/215 5:25:52 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 162 Integrating Lower-Level Perception-Action (c) No change trial Close item tested (e) Far item tested (a) Sample display Far color Close colors Contrast field Feature dimension VWM (b) Delay Feature-specific suppression via inhibitory layer Self-sustained peaks in VWM No peak in contrast field D Peaks in VWM drive same node Change Trial (d) Close item tested Peak in contrast field drives diff node S D (f) No peak in contrast field Peaks in VWM drive same node Far item tested No peak in contrast field Input Excitation Inhibition Peaks in VWM drive same node S Peaks in VWM drive same node Time FIGURE 6.7: Simulations showing enhanced change detection for metrically similar items. See text for further details. Adapted from Johnson, Spencer, Luck, and Schöner (29). Decision The means of generating change detection decisions in the three-layer model represents a refinement of an earlier version of the model used to capture position discrimination performance (see Simmering et al., 26). In the previous model, responses were generated on the basis of whether or not activation pierced a threshold within a given window of time following the onset of the test input. Specifically, a same response was generated when activation pierced threshold (due to overlap between the peak in WM and the test input), and a different response was generated when the threshold was not pierced. This means of generating decisions is limited in two ways. First, although this model is able to capture reaction times (RTs) for same responses (i.e., the time taken to pierce threshold), it cannot capture RTs for different responses because no active decision is made on these trials. Second, the approach to different responses implemented in the model is not neurally realistic because it effectively relies on the absence of activation, rather than its presence, to generate a decision. The DF model described here addresses each of these issues, making it possible to account for RTs on both change and no-change trials, and to derive specific predictions about how RTs may be expected to change under different conditions. Recall that change detection decisions arise in the model through the activation of separate response nodes that accumulate evidence supporting same versus different responses based on input from the WM and CON layers, respectively, during the comparison process described earlier. A response is generated when one of the nodes accumulates sufficient information to go above threshold (for evidence supporting the neural plausibility of this form of decision making, see review in Schall, 23). The rate at which this occurs depends on the amount of information present in CON and WM at testing. 2 2 In addition to the same and different decision nodes, the model includes a gating node that autonomously controls the f low of activation from CON and WM to the response nodes based on events in the task. This node receives direct stimulus input as well as input from WM, and only rises above threshold with sufficient activation (i.e., when test array input is presented and there are above-threshold peaks in WM). 2_med_9781993563_part_2.indd 162 8/3/215 5:25:52 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN Integrating Perception and Working Memory in a Three-Layer Dynamic Field Model 163 When strong above-threshold activation is present in one layer at testing, but not in the other, as is typically the case on no-change trials, RT is expected to be relatively fast. By contrast, when above-threshold peaks are present in both layers at testing, as is commonly the case on change trials, activation builds more slowly in the response layer as a result of competitive interactions between the nodes. From this it is expected that responses will be faster on trials where no change occurs (i.e., when a same response is made). This is exactly what has been found in experiments exploring the comparison process at the level of both perception and WM (see Farell, 1985; Hyun, Woodman, Vogel, Hollingworth, & Luck, 29). Other more subtle factors that could be expected to affect the rate of information accumulation in the response layer and, thus, RT include the number of items being held in WM and their metric similarity, as well as the magnitude of the change introduced at testing. Each of these factors could influence the level of activation present in CON and/or WM, thereby influencing the rate at which information is accumulated in the response layer at testing. For example, inhibitory interactions among peaks leads to lower levels of above-threshold activation in WM when more similar information is held, which could translate into slower responding on no-change trials when highly similar versus distinct items are remembered. C O M PA R I S O N WITH OTHER MODELS The DFT is not the only theory proposed to account for spatial recall biases and behavior in the change detection paradigm. Another prevalent account of spatial recall biases is the category adjustment model (Huttenlocher, Hedges, & Duncan, 1991). Within this framework, a given spatial location is represented with two types of information: the fine-grained location (i.e., the distance and direction of the target from a reference location) and the category in which it is located (e.g., the upper right quadrant of a page). Categories are formed by dividing the task space into regions, typically aligned with symmetry axes or visible lines. As a person remembers both fine-grained and categorical information over a short delay, the person s certainty in the fine-grained information is reduced. At recall, these two types of information are combined and weighted according to certainty; as fine-grained information is less certain, categorical information is weighted more heavily, resulting in a bias toward the prototype (i.e., center) of the category. Although this alternative approach has successfully accounted for much of the data on the direction and magnitude of bias in spatial recall, it falls short of the breadth of explanation provided by the DFT. In particular, the DFT has also addressed (a) how variability in repeated responses depends on the distance of a location from the midline symmetry axis (Spencer, Austin, & Schutte, 212); (b) how drift emerges over time to influence position discrimination performance (as discussed earlier; Simmering et al., 26); (c) how similar processes operate in memory for nonspatial features, as we described in the previous section (e.g., Johnson, Spencer, & Schöner, 29); (d) the developmental origins of these effects (see Chapter 1), and when mechanisms of Hebbian learning are added to the model; and (e) how long-term memory influences spatial recall biases (Lipinski, Simmering, Johnson, & Spencer, 21; Lipinski, Spencer, & Samuelson, 21). Thus, the DFT provides a more comprehensive account of visuospatial cognition than the category adjustment model. With respect to change detection, numerous approaches have been proposed that address one or more aspects of performance in these tasks. For example, one prominent view holds that the capacity-limited nature of WM suggested by the results of change detection experiments arises from a WM system that stores information in a limited number of discrete, fixed resolution slots (e.g., Cowan, 25; Luck & Vogel, 1997; Zhang & Luck, 28). Within the discrete slots view, performance remains high as long as the number of to-be-remembered items (i.e., the set size) is less than the number of available slots. When the set size exceeds the number of available slots, only a portion of the available information is stored in WM, and the rest of the information is simply forgotten, producing errors in the change detection task. An alternative to the discrete slots view holds that working memory is better conceived of as a shared resource that is f lexibly distributed among the elements of a visual scene (e.g., Bays & Husain, 28; Wilken & Ma, 24). As more and more items are stored in working memory, fewer and fewer resources are available to represent each one, and therefore each item is represented with decreasing precision (i.e., with greater amounts of noise). As a consequence, at high set sizes, errors arise when unchanged items in the test array are mistaken for changed items, and vice versa. Thus, within this view, there is no fixed upper limit on 2_med_9781993563_part_2.indd 163 8/3/215 5:25:52 PM

OUP UNCORRECTED PROOF FIRSTPROOFS, Mon Aug 3 215, NEWGEN 164 Integrating Lower-Level Perception-Action the amount of information that can be maintained in working memory. Instead, increasing noise as a function of set size gives the appearance of capacity limits at high set sizes in studies of change detection. The DFT shares features in common with each of these views (see Johnson, Simmering, & Buss, 214, for further detail). For example, as described earlier, the dynamic neural processes underlying maintenance in the model impart a discrete, all-or-none quality to neural representations and can give rise to capacity limits at higher set sizes. This ref lects the bistability underlying the peak state peaks either form or they do not. In contrast to the discrete slot model, however, items are not stored independently from the other items in working memory, nor are they stored with perfect fidelity. Rather, there can be metric interactions among items in memory in keeping with the resource view. Critically, the DFT goes beyond both approaches in formally specifying the processes involved in the formation, maintenance, comparison, and decision processes required in the change detection task. This feature of the model makes it possible to account for many different aspects of performance not addressed by these alternative models. For instance, because the DFT models individual change detection trials, in addition to aggregate performance, the model can account for capacity limits, and can be used to assess the factors that may contribute to the production of specific types of errors on change trials (see Johnson, Simmering, & Buss, 214). Additionally, the DFT can account for metric effects observed in studies of change detection, and can be used to derive predictions regarding factors that may inf luence the speed of encoding and response generation, as well as other aspects of performance that are out of reach for these other approaches. Importantly, the DFT is in a position to clarify how the processes underlying change detection may emerge from the complex dynamical processes supporting neural function (Buss, Magnotta, Schöner, & Spencer, 214). More generally, it is important to note that the DFT is the only theory that addresses performance in a wide variety of tasks from spatial recall to change detection and that accounts for behavior in adults as well as developmental changes in these tasks (as described in Chapter 1). Thus, the DFT provides a much richer description of behavior and cognition over development and across domains within visuospatial cognition than do competing models. CONCLUSION In this chapter, we described a three-layer neural field model that extends the DF framework described in earlier chapters to address the integration of perception and working memory in the context of recall and change detection tasks. The proposed model provides a neurally plausible account of key findings related to working memory for spatial and nonspatial visual information, and has generated several novel predictions that have been confirmed in behavioral experiments probing spatial and color recall, spatial discrimination, and change detection. More generally, the model provides a useful framework for addressing how memory representations are encoded and maintained in working memory, and how such representations are coordinated with perceptual systems to support visually guided behaviors. The ability to flexibly integrate perceptual and working memory processes is critically important in many real-world contexts, from relatively simple situations, such as comparing stimuli separated in either space or time (e.g., to pick the best apple among a bunch of apples at the store), to more complex situations, such as navigating through traffic while following a planned route. Subsequent chapters will combine this flexibility with the type of spatial and featural selection discussed in Chapter 5 as we move toward an integrated view of visual cognition in this part of the book. REFERENCES Amari, S. (1977). Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics, 27, 77 87. Amari, S., & Arbib, M. A. (1977). Competition and cooperation in neural nets. In J. Metzler (Ed.), Systems Neuroscience (pp. 119 165). New York: Academic Press. Bays, P. M., & Husain, M. (28). Dynamic shifts of limited working memory resources in human vision. Science, 321, 851 854. Buss, A. T., Magnotta, V., Schöner, G., Huppert, T. J., & Spencer, J. P. (215). Testing bridge theories of brain function with theory-derived fmri. Manuscript submitted for publication. Cowan, N. (25). Working memory capacity. Hove, East Sussex, UK: Psychology Press. Engebretson, P. H., & Huttenlocher, J. (1996). Bias in spatial location due to categorization: Comment on Tversky and Schiano. Journal of Experimental Psychology: General, 125(1), 96 18. Farrell, B. (1985). Same - different judgments: A review of current controversies in perceptual comparison. Psychological Bulletin, 98, 419 456. 2_med_9781993563_part_2.indd 164 8/3/215 5:25:52 PM