arxiv: v2 [cs.ne] 15 Sep 2017

Transcription

1 Feed-forwrd pproximtions to dynmic recurrent network rchitectures Dyln R. Muir Biozentrum University of Bsel Klingelergstrsse 5/7 Bsel 456 Switzerlnd rxiv: v [cs.ne] 5 Sep 7 (Dted: Septemer 5th 7) Recurrent neurl network rchitectures cn hve useful computtionl properties with complex temporl dynmics nd input-sensitive ttrctor sttes. However evlution of recurrent dynmic rchitectures requires solution of systems of differentil equtions nd the numer of evlutions required to determine their response to given input cn vry with the input or cn e indeterminte ltogether in the cse of oscilltions or instility. In feed-forwrd networks y contrst only single pss through the network is needed to determine the response to given input. Modern mchinelerning systems re designed to operte efficiently on feed-forwrd rchitectures. We hypothesised tht two-lyer feedforwrd rchitectures with simple deterministic dynmics could pproximte the responses of single-lyer recurrent network rchitectures. By identifying the fixed-point responses of given recurrent network we trined two-lyer networks to directly pproximte the fixed-point response to given input. These feed-forwrd networks then emodied useful computtions including competitive interctions informtion trnsformtions nd noise rejection. Our pproch ws le to find useful pproximtions to recurrent networks which cn then e evluted in liner nd deterministic time complexity. Keywords: recurrent neurl networks; fixed-point responses; feed-forwrd neurl networks INTRODUCTION With very few exceptions iologicl networks of neurons re highly recurrent. For n extreme exmple neurons in the primry visul corticl res in mmmlin rin mke mjority of their synptic connections etween other neurons in the locl vicinity (Binzegger et l. 4). Recurrent networks cn give rise to complex temporl dynmics nd potentilly eneficil emergent computtionl properties. For exmple desired reltionships etween the ctivity of severl neurons cn e emedded in recurrent excittory weights (Dougls et l. 994 Hhnloser 3 Rutishuser nd Dougls 9); the dynmics of the network cn then selectively mplify the desired representtions while rejecting noise or undesired interprettions of n input (Ben- Yishi et l. 995 Dougls nd Mrtin 7 Somers et l. 995). Chotic temporl dynmics present in reservoirs of rndomly connected neurons cn e exploited to selectively detect or generte roust temporl sequences (Lje nd Buonomno 3 Mss et l. Sussillo nd Aott 9). However simulting dynmic recurrent networks to mke use of their properties in rtificil systems is inconvenient for severl resons. Such simultions re non-deterministic in terms of the time required to find n nswer for given input. This is ecuse the dynmics of recurrent networks especilly stochsticlly-generted networks my not e gurnteed to e stle for every input nd my indeed not e known in dvnce of simultion. Even if stle fixedpoint responses exist for every finite input the time tken to rech these fixed points my differ depending on the input. This issue is excerted y the poor fit etween simultions of recurrent networks nd commodity computtionl rchitectures (i.e. CPU / GPU). Author s finl version ccepted for puliction in Neurl Computtion. dyln.muir@unis.ch; In contrst recent successes in using feed-forwrd or unrolled recurrent rchitectures (Grves et l. 3 Rdford et l. 5) hve occurred hnd in hnd with development of computtionl systems optimised for evlution of feedforwrd networks (Colloert et l. Ji et l. 4 Theno Development Tem 6). Modern pproches for distriuted evlution of lrge networks (Adi et l. 6) mke feed-forwrd rchitectures very ttrctive for rnge of pplied computtionl tsks. Here we exmine whether the known eneficil computtionl properties of highly recurrent network rchitectures cn e relised in feed-forwrd rchitectures. We tke the pproch of proing recurrent networks to quntify mpping etween inputs nd fixed-point responses. We then trin feed-forwrd networks to pproximte this mpping nd compre the informtion-processing ilities of the recurrent networks with their feed-forwrd pproximtions. RESULTS Recurrent networks nd feed-forwrd pproximtions Fig. shows n exmple of simple -neuron single-lyer recurrent network. The dynmics of ech rectified-liner (or liner-threshold; or ReLU) neuron (x j composed into vector of ctivity x) is governed y nonliner differentil eqution τ x + x = W R [x] + +i () (see Methods) nd evolves in response to the input i provided to the neuron s well s the ctivity of the rest of the network x trnsformed y the recurrent synptic weight mtrix W R. Here [x] + denotes the liner-threshold trnsfer function [x] + = mx(x). Neglecting the potentilly complex temporl dynmics of network ctivity for this

2 i x WR [f ]+ WFF x WFF x i Figure. Recurrent nd feed-forwrd network rchitectures. () Two-neuron recurrent rchitecture. Rectified-liner (ReLU) neurons (x; circles) receive input (i) nd possily rech stle fixed point in ctivity (f; the vlues of network ctivity x t the fixed point if it exists) through recurrent interctions vi weights W R. () A neuron feed-forwrd rchitecture. Input i is trnsformed through two lyers of ReLU neurons (x x ) vi ll-to-ll weight mtrices WF F nd WF F. The ctivity x of lyer is the output of the network. See Methods for more detil. Smll network rchitectures We first investigted whether the dynmics of -neuron recurrent networks cn e pproximted y trining twolyer liner-threshold neuron (ReLU) network to directly mp network inputs to fixed-point responses of the recurrent network. We otined ccurte feed-forwrd pproximtions for rndomly chosen neuron recurrent networks tht exhiited stle non-trivl fixed points. Here we show two exmples of networks with oth non-oscilltory nd oscilltory dynmics. Fig. shows the result of pproximting -neuron recurrent network with positive rel eigenvlues which led to stle fixed points with n expnsive mpping of the input spce. We performed rndom smpling of the input spce y drwing uniform rndom vrites from the unit squre ( ). For ech input we nlysed the eigenspectrum nd solved the dynmics of the recurrent network to determine whether stle fixed-point response existed for tht input discrding inputs for which no stle fixed point existed. - xɩ xɩ xɩ - xɩ Figure. Feed-forwrd pproximtion to the fixed-point mpping of neuron recurrent network with rel positive eigen.4.. vlues. () Recurrent dynmics for the system W R =.8.5 Loci of recurrent network responses trced to fixed points (dots) from mtrix of inputs I rrnged uniformly over the unit squre ( ) (solid squre). Ech line trces the locus of x in response to single input im to the corresponding fixed point fm. The origin is indicted y the lck cross. () Rectified fixed-point responses F 3 [fm ]+ of the recurrent network (circles) overlid with the corresponding feed-forwrd network response (dots). Ornge lines connect poorly-mpped feed-forwrd responses to the corresponding recurrent fixed point. WF F WF F F F F F = ] 3 -[ In the following we pproximte the mpping etween network inputs i nd network fixed points [f]+ using fmily of feed-forwrd network rchitectures (Fig. ). For recurrent network with N = neurons the corresponding feed-forwrd pproximtion consisted of two lyers ech consisting of N = ReLU neurons. All-to-ll weight mtrices WF F nd WF F defined the connectivity etween the network input (i) the neurons of lyer (x ) nd the neurons of lyer (x ). We use the nottion v n to refer to vrile v within lyer n. In some implementtions of unrolled recurrent network rchitectures weight mtrices cross severl lyers representing multiple points in time re tied together nd trined s group. We did not tke tht pproch with our feed-forwrd networks nd permitted the weights for ech lyer to vry independently. The ctivity of x were tken s the output of the network. In contrst to the recurrent network neuron ctivtions in the feed-forwrd pproximtion were given y deterministic feed-forwrd evlution with no temporl dynmics (Eq. 5; see Methods). Error s(x - [ f m ] + ) work we define the result of such network s the rectified fixed-point response of the popultion ctivity of the network [f ]+ if stle fixed point exists. 4 Fixed point min[fm] Fixed point min[fm]+ Figure 3. Errors in the feed-forwrd pproximtion occur minly round the ctivtion threshold. () Lrge differences etween the output of the feed-forwrd pproximtion (x ) nd the rectified fixed-point of the recurrent network ([fm ]+ ) occur mostly when the ctivtion of one recurrent unit is elow threshold (i.e. x x < ; [fm ]+ = ). () Errors in generliztion increse s the input to the network moves further outside the trined region (ι ι > ) ut remin smll. We therefore found mpping etween set of inputs I nd the set of corresponding fixed-point responses F which ws used s trining dt to find n optiml feed-forwrd pproximtion to tht mpping (see Methods). Fig. shows the ctivity dynmics of the recurrent network from numer of inputs to their corresponding fixed points.

3 3 A B Exc Inh Response 3 B A A B Input to prtitions Recurrent fixed point response Feed-forwrd pproximtion xɩ xɩ - xɩ - B Figure 5. Competition in two-prtition network. () A recurrent dynmic network with two ll-or-nothing excittory prtitions (A nd B) nd single glol inhiitory neuron (Inh). The rchitecture of the feed-forwrd pproximtion network is s shown in Fig.. () Stimulting the prtitions with liner mixture of input currents ( ) (grey shding) provokes strong competition in the response of the prtitions (note rpid switching etween prtition A nd B when inputs re roughly equl). The feed-forwrd pproximtion (dshed lines) exhiits similr competitive switching to the recurrent network (solid lines). Recurrent network prmeters: {N we w I } = {5.5 8 }. A xɩ Figure 4. Feed-forwrd pproximtion to neuron recurrent network with dmped oscilltory dynmics nd complex eigenvlues. () Recurrent dynmics for the.7. system W R =. This network hs complex ei genvlues with dmped oscilltory dynmics. () Recurrent fixed-point responses (circles) compred with feed-forwrd network responses (dots). WF F WF F F F F F = Nottions s in Fig.. We used stochstic grdient-descent optimistion lgorithm with momentum nd dptive lerning rtes (Adm; Kingm nd B 5) to find feedforwrd network tht pproximted the mpping I F y minimising the men-squre loss function c = + /M Σ M (see Methods). The Adm optimm= xm [fm ] istion lgorithm resulted in feed-forwrd pproximtions with smller errors thn trining using direct grdient descent without momentum. Only fixed points in which ll elements fm > were used for trining. We found this pproch to result in etter pproximtions to recurrent fixed points. Since mny inputs mp to zero fixed point responses in the recurrent network (see Fig. ) the trining process tended to over-emphsise them leding to poor representtion of non-zero fixed points. Trining ws performed over rndomly generted tches contining M = 5 input to fixed-point mppings nd ws hlted when the tch trining error smoothed over tches converged. Fig. shows the I F mpping produced y the est feed-forwrd network found fter 6 5 trining itertions. Inputs tht led to non-zero response from oth neurons were mpped with high ccurcy (overlpping dots nd circles). Errors in the feed-forwrd pproximtion occurred minly round the ctivtion threshold (Fig. 3). The feed-forwrd pproximtion lso generlized well for inputs outside the trining regime (i.e. ι ι > ; Fig. 3). Generliztion errors incresed slowly further from the trined input spce ut remined smll. In this exmple we were therefore le to trin n ccurte feed-forwrd pproximtion to the fixed-point dynmics of this simple recurrent network. How does this pproch fre when pplied to recurrent network with more complex dynmics? Fig. 4 shows the result of pproximting -neuron recurrent network with complex eigenvlue pir with positive rel prt which leds to stle spirl fixed points. This recurrent network exhiited dmped oscilltory dynmics when driven y constnt inputs (Fig. 4). Nevertheless our pproch of pproximting the mpping I F ws successful. Fig. 4 shows comprison etween the recurrent nd feed-forwrd network mppings. As efore errors in the feed-forwrd pproximtion were restricted to the re round the neuron ctivtion threshold. Our pproch is therefore le to find feed-forwrd pproximtions to recurrent networks with complex temporl dynmics. Competitive networks with prtitioned excittory structure There is growing evidence for network rchitectures in cortex tht group excittory neurons into soft-prtitioned sunetworks (Ko et l. Yoshimur et l. 5). Connections within these sunetworks re stronger nd more prevlent (Cossell et l. 5). Sunetwork memership my e defined y response similrity; neurons with correlted responses over long periods will therefore tend to e connected (Cossell et l. 5 Ko et l. Lee et l.

4 4 6). These rules for connection proility nd strength cn give rise to network rchitectures with complex dynmicl nd stility properties including selective mplifiction nd competition etween prtitions (Muir nd Mrsic-Flogel 5). We investigted simplified version of sunetwork prtitioning with ll-or-nothing recurrent excittory connectivity (Fig. 5; see exmple mtrix in Methods). Networks with this connectivity pttern exhiit strong recurrent recruitment of excittory neurons within given prtition coupled with strong competition etween prtitions medited y shred inhiitory feedck. As consequence the recurrent network cn e viewed s solving simple clssifiction prolem wherey the network signls which is the greter of the summed input to prtition A (ι + = ι + ι ) or to prtition B (ι 3+4 = ι 3 + ι 4 ). In ddition the network signls n nlogue vlue linerly relted to the difference etween the inputs. If ι + > ι 3+4 then the network should respond y strong ctivtion of x nd complete inctivtion of x 34 (nd vice vers for ι + < ι 3+4 ). We first exmined the strength of competition present etween excittory prtitions y providing mixed input to oth prtitions compring the recurrent network response with the feed-forwrd pproximtion (Fig. 5). Input ws provided eqully to oth excittory neurons in prtition such tht ι = ι nd ι 3 = ι 4. For given network evlution single mixture ws chosen such tht i ws constnt. When input to prtition A ws wek input to prtition B ws strong nd vice vers. The recurrent network ws permitted to rech stle fixed point for given sttic input mixture nd the feed-forwrd pproximtion ws evluted with the sme input pttern. The recurrent network exhiited strong competition etween responses of the two excittory prtitions: only single prtition ws ctive for given network input even when the input currents to the two prtitions were lmost equl. The feed-forwrd pproximtion exhiited very similr competition etween responses of the two prtitions s the recurrent network lso exhiiting shrp switching etween the prtition responses (Fig. 5). In ddition the feed-forwrd network lerned good pproximtion to the nlogue response of the recurrent network s for the simpler networks of Figs 4. Although the feed-forwrd pproximtion ws not trined explicitly s clssifier we exmined the extent to which the feed-forwrd pproximtion hd lerned the decision oundry implemented y the recurrent network (Fig. 6). Multi-lyer feed-forwrd neurl networks of course hve long history of eing used s clssifiers (e.g. LeCun et l. 989 Rumelhrt et l. 986). The purpose of the pproch presented here is to exmine how well the feed-forwrd pproximtion hs lerned to mimic the oundries etween sins of ttrction emedded in the recurrent dynmic network. This question is prticulrly interesting for lrger nd more complex recurrent networks for which the oundries etween sins of ttrction re not known priori. We exmined the response of the feed-forwrd pproximtion close to the idel decision oundry (ι + = ι 3+4 ; dshed line in Fig. 6). We found tht the mjority of inputs were correctly clssified y the feed-forwrd pproximtion ut the decision oundry of the feed-forwrd pproxim Correct Misclssifi ed Decision oundry Figure 6. Decision oundry is lmost ligned etween recurrent network nd feed-forwrd pproximtion. Shown is the projection of the input spce i into two dimensions ι + = ι + ι nd ι 3+4 = ι 3 + ι 4. The idel decision oundry of ι + = ι 3+4 is indicted s dshed horizontl line. The mjority of inputs smpled close to the decision oundry resulted in the ctivtion of the correct prtition in oth the recurrent network nd feedforwrd pproximtion (grey dots). The decision oundry of the feed-forwrd pproximtion ws not perfectly ligned with tht of the recurrent network resulting in misclssifiction of some inputs close to the decision oundry (ornge dots). Network prmeters s in Fig. 5. tion ws not perfectly ligned with the idel with the result tht minority of inputs close to the oundry were misclssified y the feed-forwrd pproximtion. Line ttrctor networks Neurons in primry visul cortex of primtes nd crnivores hve individul preferences for the orienttion of line segment in visul spce (Huel nd Wiesel ); neurons tht prefer similr orienttions re grouped together nd this preference chnges smoothly cross the surfce of cortex (Blsdel 99 Bonhoeffer nd Grinvld 99). Experimentl work suggests tht the shrp tuning of visul neurons for their preferred orienttion rises through recurrent processing within the corticl network (Tsumoto et l. 979) rther thn eing defined y structured inputs to ech neuron from outside the locl network (Huel nd Wiesel 96). The recurrent processing hypothesis is lso consistent with the fct tht the mjority of input synpses to ech neuron rise from other nery neurons nd not from visul input pthwys (Ahmed et l. 994 Binzegger et l. 4 Peters nd Pyne 993). Severl recurrent network models of mmmlin cortex mke use of the fct tht the function of neurons chnges smoothly cross the surfce of mny corticl res (Ben- Yishi et l. 995 Dougls et l. 994 Somers et l. 995). The tight reltionship etween physicl nd functionl spce (i.e. the preferred orienttion θ of neuron) suggests tht locl neuronl connections should e mde predomintely etween neurons with similr θ flling off with distnce. In these recurrent models excittory neurons re consequently rrnged in ring (therefore ring models ; Fig. 7) with smoothly-vrying θ nd with excittory connection strength flling off with decresing similrity in θ. Inhiitory neurons re rodly tuned or untuned for preferred orienttion in these models nd therefore mke nd receive connections with ll excittory neurons. These ring models perform powerful nd useful informtion processing tsks which re supported y mechnisms of selective mplifiction through recurrent excittion coupled with competitive interctions medited y glol inhiit-

5 5 W R.5 c Inh W FF W FF Input tuning width Figure 7. Feed-forwrd pproximtion recovers the weight structure of recurrent ring model for orienttion preference. () A schemtic of ring model for orienttion preference. Excittory neurons (outer ring) re rrnged in order of preferred orienttion (θ). Recurrent excittory connections upper rrows re modulted y similrity in θ (see Methods). Inhiitory connections (from centrl neuron; Inh) re mde with ll excittory neurons. The rchitecture of the corresponding feed-forwrd pproximtion network is s shown in Fig.. Recurrent excittory connections re shown for single neuron ut re mde identiclly from ll excittory neurons. () Recurrent weights W R implementing the ring model in dynmic recurrent network. Inhiitory weights were wekened y fctor of for visulistion. (c) Weights lerned in feed-forwrd pproximtion to the dynmics of recurrent ring model. Note the neighourhood pttern lerned y the feed-forwrd network similr to tht of the recurrent network. Recurent weights W R nd second-lyer weights W FF were scled for visulistion purposes. Recurrent network prmeters: {N w e w i } = {45}. ory feedck (lso known s winner-tke ll interctions; Dougls nd Mrtin 7). Single neurons exhiit consistent shrp tuning for their preferred orienttion θ in spite of poorly-tuned input. Ring networks re lso le to reject significnt noise in the input to provide clen interprettion of noisy signl. Recurrent dynmics within the network estlish line ttrctor wherey set of stle response ptterns tht re trnslted versions of common ctivity pttern re permitted y the network. We investigted whether feed-forwrd pproximtion to simple ring model for orienttion preference could cpture useful informtion-processing fetures of the recurrent network. We trined two-lyer neuron network to pproximte the fixed-point recurrent dynmics of 4- neuron recurrent ring model network (see Methods). We generted the trining mpping I F y generting uniform rndom inputs i m U (.5) nd solving the dynmics of the recurrent network to identify the corresponding fixed points f m (see Methods). We discrded inputs for which no corresponding fixed point could e found. Fig. 7 shows the weight mtrices for the two-lyer network est pproximting the recurrent dynmics fter 64 - Network input Recurrent fi xed point response Feed-forwrd pproximtion Figure 8. Shrpening of rodly-tuned input. Shown re polr plots of preferred orienttion θ versus network response mplitude for given input (lck dshed) for oth the recurrent model (lue) nd the feed-forwrd pproximtion (ornge). Rows correspond to four exmples ech of input under incresing tuning shrpness κ (indicted t left) nd rndomly-chosen Θ (lck line in ech exmple). Noise std. dev. ζ = for these exmples; common-mode input γ =.5. All plots hve identicl scling. Note the similrity etween recurrent fixed point responses nd the feed-forwrd pproximtion nd the consistency in response tuning over rnge of rodly-tuned inputs ptterns. Network prmeters s in Fig. 7. trining itertions. Note tht the neighourhood reltionships etween similrly-tuned neurons is reflected in the lerned feed-forwrd weight structure which hs een cquired solely y mimicking the fixed-point dynmics of the recurrent network (c.f. Fig. 7c). The loclity of mpping etween djcent neuron indices ws encourged y initilising the feed-forwrd weights W FF nd W FF to the identity mtrix I t the eginning of trining (see Methods). The ring model ws designed to demonstrte how recurrent processing cn led to shrpening of rodly-tuned inputs. To investigte whether our feed-forwrd pproximtion exhiits similr functionlity we stimulted recurrent nd feed-forwrd networks with rodly-tuned inputs (Fig. 8). Indeed the responses of the feed-forwrd pproximtion were shrpened versions of the input nd hd similr tuning shrpness s the recurrent network. Interestingly the shrpness of response tuning of the feed-forwrd network did not chnge pprecily cross wide rnge of input tuning shrpnesses. The feed-forwrd pproximtion ws therefore le to cpture the min informtion-processing feture of the recurrent ring model. Noise rejection in the recurrent ring model is medited y recurrent shped excittory mplifiction of responses coupled with glol inhiitory feedck. We investigted whether the feed-forwrd pproximtion ws le to perform equivlent noise rejection in the sence of recurrent excittory mplifiction. We stimulted the network with tuned inputs with incresing mounts of Normllydistriuted noise with std. dev. ζ (Figs 9 nd ; see Methods).

6 6 Input noise std. dev....3 Network input Recurrent fi xed point response Feed-forwrd pproximtion Figure 9. Feed-forwrd pproximtions cn mimic the noise rejection properties of the recurrent ring model. Rows correspond to four exmples ech of input under incresing noise std. dev. ζ (indicted t left) nd rndomly-chosen Θ. Tuning shrpness κ = 4; common-mode input γ =.5. Network prmeters nd nottions s in Figs 7 nd 8. Error ( ) 45 Θ- R R - FF error in stimulus interprettion etween the recurrent nd feed-forwrd networks θ R θ FF (Fig. lck). This error ws consistently clustered round zero highlighting the closeness of the feed-forwrd pproximtion to the ehviour of the recurrent model (see lso exmples in Fig. 9). As expected the ility of oth models to correctly identify the underlying stimulus orienttion Θ degrded with incresing noise mplitude ζ (incresing errors Θ θ R Fig. lue). The men error etween the response of the recurrent network nd the feed-forwrd pproximtion men ( s { x [f m ] +}) lso incresed with incresing noise mplitude ζ (Figs 9 nd ). The recurrent ring models perform common-mode input rejection wherey the response of the recurrent dynmic network is unchnged y dding common-mode offset to n input. This occurs through dynmic thresholding of the network response provided y glol inhiitory feedck. Our feed-forwrd pproximtions were trined with fixed common-mode input γ (see Methods). We exmined the ility of the feed-forwrd pproximtions to generlise their responses given ritrrily-scled common mode input (Fig. ). For feed-forwrd pproximtions trined with γ =.5 we found tht solute pproximtion errors remined low for γ. (i.e. error mplitudes <.). For lrger γ errors scled linerly with the response of the feed-forwrd network indicting tht the pproximtion reks down. This result suggests tht mtching of the input spce to the trining spce is required for ccurte pproximtion either through pproprite selection of trining inputs I or through input normlistion. Men s(x -[f m ] + ) Noise std. dev..5 Noise std. dev. Figure. Noise rejection y the feed-forwrd pproximtion is roust over rnge of noise mplitudes. () The error etween the recurrent model ngle of pek response θ R nd the feed-forwrd pproximtion ngle of pek response θ FF (lck) ws consistently smll over ll noise mplitudes ζ. For incresing noise mplitudes the ility of the recurrent model to correctly identify the orienttion of the input θ ι degrded (lue). () The ccurcy of the feed-forwrd pproximtion with respect to the recurrent model degrded grdully with incresing noise mplitude ζ. Tuning shrpness κ = 4; common-mode input γ =.5. Network prmeters s in Fig 7. We quntified the error in network responses in severl wys. Firstly the purpose of noise rejection in the recurrent model is to identify the orienttion Θ of the underlying stimulus. We defined the ngle of pek response θ R s the preferred orienttion θ of the neuron with pek response i.e. θ j : j = rgmx x j nd defined θ FF nlogously for the feed-forwrd pproximtion. We then quntified the DISCUSSION We investigted whether feed-forwrd neurl networks could pproximte the fixed-point responses of dynmic recurrent networks. We trined two-lyer feed-forwrd rchitectures to replicte the input-to-fixed-point mpping of dynmic recurrent networks. We found tht for smll ritrry networks lrger networks with prtitioned excittory nd inhiitory neurons nd multiple prtitioned excittory popultions s well s even lrger networks emedding line ttrctors two-lyer feed-forwrd pproximtions were le to successfully reproduce the fixed-point responses of dynmic recurrent networks. Feed-forwrd pproximtions reproduced the fixed-point responses for two-neuron dynmic recurrent networks for recurrent networks with oth simple nd complex temporl dynmics (Figs nd 4). In the cse of dynmic recurrent network exhiiting competitive interctions etween excittory prtitions the feed-forwrd pproximtion ccurtely replicted competition etween prtitions (Fig. 5). Our pproch ws le to find good pproximtion to line ttrctor network with highly nonliner dynmics soft winner-tke-ll ring model for preferred orienttion. This ws impressive considering tht the trining inputs provided to the network were uniformly rndomly distriuted nd did not tke into ccount the line ttrctor computtion performed y the recurrent network. The feed-forwrd pproximtion reproduced nonliner input trnsformtions nd noise rejection oth of which re considered to e

7 We found tht the ccurcy of the pproximtions degrded close to the ctivtion thresholds of the feed-forwrd network (Fig. 3). This my e due to the hrd loss of grdient informtion elow the ctivtion threshold in which cse using units with soft nonlinerity might llevite this issue. However the feed-forwrd pproximtions to the two-neuron recurrent networks generlized well for inputs outside the trined input spce (Fig. 3) with errors incresing slowly ut remining low for inputs well outside the trining regime. Common mode input mplitude Network input Recurrent fi xed point response Feed-forwrd pproximtion Figure. Common-mode input rejection fils for lrge common-mode mplitudes. Shown re exmples of the recurrent network (lue) nd the feed-forwrd pproximtion responses (ornge) when driven with inputs with vrying common-mode input mplitudes γ (lck dshed). The feed-forwrd pproximtion ws trined with γ =.5. While the recurrent dynmic network rejected common mode inputs over ll mplitudes the feed-forwrd pproximtion cesed to perform well for γ >. For lrge γ common-mode noise ws not rejected y the feed-forwrd pproximtion (see Fig. ). Error x -[f m ] FF resp. x Common mode input mplitude Figure. The feed-forwrd pproximtion cnnot reject common-mode input of ritrry mplitude. Shown is the error etween the feed-forwrd pproximtion response x nd the recurrent dynmic network response [f m ] + to given input s function of the scle of the feed-forwrd response x for rnge of common mode input mplitudes γ. The feed-forwrd pproximtion ws trined with γ =.5. Errors for γ < were clustered close to zero. Network prmeters s in Fig. 7. prticulrly useful fetures of recurrent computtion in the model (Figs 8 ). The feed-forwrd pproximtion to the recurrent ring model generlized well for inputs up to fctor of 4 outside the trining regime (Fig. ). However the pproximtion roke down for inputs with lrger mplitudes in spite of the liner trnsfer functions present in ech neuron. Nevertheless this restriction simply entils the use of normlised input spces to ensure ccurcy of the pproximtion. Our feed-forwrd pproximtions implicitly ssume tht only the fixed-point response to n input is importnt nd the temporl evolution of ctivity to rech tht fixed point is ignored. Some modes of opertion of dynmicl recurrent networks explicitly mke use of chotic dynmics to detect nd generte temporl ctivity sequences (Lje nd Buonomno 3 Mss et l. Sussillo nd Aott 9). Computtions tht require ccess to ctivity trjectories will of course not e possile under the frmework we proposed here. An pproch might e possile where network ws trined with step-wise pproximtions to the dynmics of recurrent network ut the purpose of our pproximtions ws to ovite the use of itertive solutions. Since our feed-forwrd networks hve no temporl dynmics they lso cnnot cpture complex dynmicl ehviours such s dmped oscilltory or limit cycle dynmics (Lndsmn et l. ). The response of the feed-forwrd pproximtions to given input does not depend on previous network ctivity in the formultion presented here. Responses to temporl input sequences will therefore only e ccurte if the time constnt of input chnges is much slower thn the time constnt of the dynmics of the originl recurrent network nd if complex sins of ttrction re not present. Relted to this point unrolled recurrent rchitectures such s LSTM networks hve een employed to process discrete temporl input sequences (Hochreiter nd Schmidhuer 997 Liwicki et l. 7). Our feed-forwrd pproximtions could e operted in similr mode y ugmenting the current input i(t) with the previous fixed-point ctivity x (t ). Feed-forwrd pproximtions to dynmic recurrent systems re powerful tool for cpturing the informtion processing enefits of highly recurrent networks in conceptully nd computtionlly simpler rchitectures. Informtion processing tsks such s selective mplifiction nd noise rejection performed y recurrent dynmicl networks cn therefore e incorported into feed-forwrd network rchitectures. Evlution of the feed-forwrd pproximtions is deterministic in time in contrst to seeking fixed-point response in the dynmic recurrent network where the time tken to rech fixed-point response nd indeed the existence of stle fixed point cn depend on the input to the network. Feed-forwrd pproximtions provide gurnteed solution for ech network input lthough in the cse of oscilltory or unstle dynmics in the recurrent network the pproximtion will e inccurte. Finlly the rchitecture of the feed-forwrd pproximtions is comptile with modern systems for optimised nd distriuted evlution of deep networks.

8 8 METHODS Dynmic recurrent networks We exmined dynmic networks of fully recurrently connected liner-threshold (rectified-liner; ReLU) neurons. ReLU neurons pproximte the firing-rte dynmics of corticl neurons (Ermentrout 998); cn e mpped idirectionlly to spiking neuron models (Neftci et l. 3 Shriki et l. 3); nd hve een pplied successfully in lrge-scle mchine lerning prolems (Glorot et l. ). The ctivity of neurons in the network evolved under the dynmics τ x + x = W R [x ] + + i () where x = { x x... x j... x N } T is the vector of ctivtions of ech neuron j; N is the numer of neurons in the network; W R N N is weight mtrix defining recurrent connections within the network; R N is the vector of neuron ises; i R N is the vector of constnt inputs to ech neuron in the network; τ is the time constnt of the neurons; nd [x] + is the liner-threshold trnsfer function [x] + = mx(x). Without loss of generlity in this work we took = nd τ = for the dynmic recurrent networks. Recurrent network fixed points Fixed points in response to given input i were defined s those non-trivil vlues for x such tht τ x =. We solved the system of differentil equtions Eq. using Runge-Kutt (45) solver. With constnt input provided from t = nd with x t= = i if no fixed-point solution ws found etween t = ( 6) then the corresponding input ws ndoned. We lso ndoned the serch if the current ctive prtition (i.e. the set of neurons with ctivity > nd their ssocited weights) hd n eigenvlue λ + with lrgest rel prt > nd the corresponding eigenvector v + hd ll positive elements (Hhnloser 998) indicting unstle network ctivity for which no stle fixed-point would e reched. For given input i m we denote the corresponding fixed point of recurrent dynmics s f m. Feed-forwrd pproximtions were trined to mtch the rectified ctivity of ech neuron [f m ] +. mmmlin corticl networks (Cossell et l. 5 Ko et l. Yoshimur et l. 5). An exmple weight mtrix is given y w E w E w I w E w E w I W R = w E w E w I w E w E w I w E w E w E w E w I (3) where {w E w I } = {4} nd unlelled entries of W R re zero. Networks with this structure cn exhiit coopertion etween neurons within single prtition nd competition etween neurons in differing prtitions. Networks with emedded line ttrctors In this pper we implemented version of the clssicl model for orienttion tuning (Ben-Yishi et l. 995 Dougls et l. 994 Somers et l. 995) where recurrent mplifiction nd competition opertes on wekly-tuned inputs to produce shrply-tuned network responses. A schemtic network with the rchitecture descried elow is shown in Fig. 7. Excittory neurons were rrnged round ring numered j = ( N ). Ech neuron ws ssigned preferred orienttion θ in order round the ring with θ j = ( π π). Recurrent excittory connection strength ws modulted y similrity of preferred orienttion. The symmetric connections etween neurons i nd j for i j = ( N ) were given y w ji = mx(cos[θ θ ]). Excittory recurrent weights were normlised such tht Σi= N w ji = w E. Excittory to inhiitory weights re given y w N j = j = ( N ). Inhiitory weights were given y w j N = w I /N with j = ( N). Input ws provided to neurons round the ring using von Mises-like function given y ι j = mx { exp [ κ cos ( θ j Θ )] + γ + z j } (4) where j = ( N ); Θ is the nominl orienttion represented y given input pttern; κ is distriution prmeter tht determines the shrpness of the input where κ = corresponds to uniform input nd lrge κ corresponds to shrply-tuned input; γ is common-mode input term (γ =.5 for trining); nd z j re Normlly-distriuted frozen noise vrites with std. dev. ζ such tht z j N ( ζ). Input to the inhiitory neuron j = N ws zero i.e. ι N =. Recurrent network rchitectures Feed-forwrd network rchitecture Rndom networks We generted numer of rndom network rchitectures y choosing W R where weights w ji re uniformly distriuted with w ji U ( ) nd =. We discrded ny systems for which no stle fixed points could e found. Two exmples for N = re shown in Figs 4. Networks with modulr prtition structure We exmined networks such tht columns of W were either excittory or inhiitory following rchitectures designed to e similr to mmmlin corticl neuronl networks (Dwivedi nd Jln 3 Rjn nd Aott 6 Wei ). We defined these networks to hve modulr or plnted prtition sunetwork structure in the excittory popultion (Muir nd Mrsic-Flogel 5) inspired y connectivity ptterns in We trined two-lyer feed-forwrd liner-threshold (ReLU) networks. The response of the network ws given y x = [ W FF i FF ] + (5) x = [ W FF x FF ] +. (6) The nottion v n indictes vrile v within lyer n of feedforwrd network. Feed-forwrd networks were trined to pproximte the fixed-point responses of given recurrent rchitecture. A set of rndom inputs I i m ws generted nd mpping found to the set of corresponding fixed-point responses F [f m ] + with fixed points found s descried ove. Inputs for which corresponding fixedpoint could not e found were discrded.

9 9 The network feed-forwrd weights { W FF W FF } nd neuron ises { FF FF } were trined using the Adm optimiser stochstic grdient descent lgorithm incorporting dptive lerning rtes nd momentum on individul model prmeters (Kingm nd B 5) with met-prmeters set s {α β β ɛ} = { }. The network ws optimised to minimise the men-squre loss function c = /M M m= ( xm [f m ] +). Anlyticl prmeter grdients were clculted using ckpropgtion of errors; zero grdients were replced with smll Normlly-distriuted rndom vlues N ( 5). Initil vlues for trining were set to the identity mtrix plus smll-mgnitude uniform rndom vrites such tht { W FF W FF } = Id(N) + U ( ) ; ises were initilised to { FF FF } =.. The Mtl implementtion of the Adm optimiser used in this work is ville from ACKNOWLEDGEMENTS The uthor thnks S Sdeh M Cook nd F Roth for helpful discussions. Adi M. Brhm P. Chen J. Chen Z. Dvis A. Den J. Devin M. Ghemwt S. Irving G. Isrd M. Kudlur M. Levenerg J. Mong R. Moore S. Murry D. G. Steiner B. Tucker P. Vsudevn V. Wrden P. Wicke M. Yu Y. nd Zheng X. (6). Tensorflow: A system for lrge-scle mchine lerning. In Proceedings of the th USENIX Conference on Operting Systems Design nd Implementtion OSDI 6 pges Berkeley CA USA. USENIX Assocition. Ahmed B. Anderson J. C. Dougls R. J. Mrtin K. A. C. nd Nelson J. C. (994). Polyneuronl innervtion of spiny stellte neurons in ct visul cortex. Journl of Comprtive Neurology 34(): Ben-Yishi R. Br-Or R. L. nd Sompolinsky H. (995). Theory of orienttion tuning in visul cortex. Proc Ntl Acd Sci U S A 9: Binzegger T. Dougls R. J. nd Mrtin K. A. C. (4). A quntittive mp of the circuit of ct primry cortex. Journl of Neuroscience 4(39): Blsdel G. G. (99). Differentil imging of oculr dominnce nd orienttion selectivity in monkey strite cortex. Journl of Neuroscience (8): Bonhoeffer T. nd Grinvld A. (99). Iso-orienttion domins in ct visul cortex re rrnged in pinwheel-like ptterns. Nture 353: Colloert R. Kvukcuoglu K. nd Fret C. (). Torch7: A mtl-like environment for mchine lerning. In BigLern NIPS Workshop. Cossell L. Icruso M. F. Muir D. R. Houlton R. Sder E. N. Ko H. Hofer S. B. nd Mrsic-Flogel T. D. (5). Functionl orgniztion of excittory synptic strength in primry visul cortex. Nture 58(7539): Dougls R. J. Mhowld M. A. nd Mrtin K. A. C. (994). Hyrid nlog-digitl rchitectures for neuromorphic systems. IEEE Interntionl Conference on Neurl Networks 3: Dougls R. J. nd Mrtin K. A. C. (7). Recurrent neuronl circuits of the neocortex. Current Biology 7(3):R496 R5. 5 Dwivedi S. K. nd Jln S. (3). Extreme-vlue sttistics of networks with inhiitory nd excittory couplings. Physicl Review E 87(474): Ermentrout B. (998). Lineriztion of f-i curves y dpttion. Neurl Computtion : Glorot X. Bordes A. nd Bengio Y. (). Deep sprse rectifier neurl networks. In Gordon G. J. nd Dunson D. B. editors Proceedings of the Fourteenth Interntionl Conference on Artificil Intelligence nd Sttistics (AISTATS-) volume 5 pges Journl of Mchine Lerning Reserch - Workshop nd Conference Proceedings. 8 Grves A. r. Mohmed A. nd Hinton G. (3). Speech recognition with deep recurrent neurl networks. In 3 IEEE Interntionl Conference on Acoustics Speech nd Signl Processing pges Hhnloser R. H. R. (998). On the piecewise nlysis of networks of liner threshold neurons. Neurl Networks : Hhnloser R. H. R. (3). Permitted nd foridden sets in symmetric threshold-liner networks. Neurl Computtion 5: Hochreiter S. nd Schmidhuer J. (997). Long short-term memory. Neurl computtion 9(8): Huel D. H. nd Wiesel T. N. (96). Receptive fields inoculr interction nd functionl rchitecture in the ct s visul cortex. Journl of Physiology (London) 6: Huel D. H. nd Wiesel T. N. (968). Receptive fields nd functionl rchitecture of monkey strite cortex. Journl of Physiology (London) 95: Ji Y. Shelhmer E. Donhue J. Kryev S. Long J. Girshick R. Gudrrm S. nd Drrell T. (4). Cffe: Convolutionl rchitecture for fst feture emedding. rxiv preprint rxiv: Kingm D. P. nd B J. (5). Adm: A method for stochstic optimiztion. In Interntionl Conference on Lerning Representtions (ICLR). 3 9 Ko H. Hofer S. B. Pichler B. Buchnn K. A. Sjöström P. J. nd Mrsic-Flogel T. D. (). Functionl specificity of locl synptic connections in neocorticl networks. Nture 473:

10 Lje R. nd Buonomno D. V. (3). Roust timing nd motor ptterns y tming chos in recurrent neurl networks. Nture Neuroscience 6(7): Lndsmn A. Neftci E. nd Muir D. R. (). Noise roustness nd sptilly-ptterned synchronistion of corticl network oscilltors. New Journl of Physics 4():33. 7 LeCun Y. Boser B. Denker J. S. Henderson D. Howrd R. E. Hurd W. nd Jckel L. D. (989). Bckpropgtion pplied to hndwritten zip code recognition. Neurl Computtion (4): Lee W.-C. A. Bonin V. Reed M. Grhm B. J. Hood G. Glttfelder K. nd Reid R. C. (6). Antomy nd function of n excittory network in the visul cortex. Nture. 3 Liwicki M. Grves A. Bunke H. nd Schmidhuer J. (7). A novel pproch to on-line hndwriting recognition sed on idirectionl long short-term memory networks. In Proc. 9th Int. Conf. on Document Anlysis nd Recognition volume pges Mss W. Ntschläger T. nd Mrkrm H. (). Rel-time computing without stle sttes: A new frmework for neurl computtion sed on perturtions. Neurl Computtion 4: Muir D. R. nd Mrsic-Flogel T. (5). Eigenspectrum ounds for semirndom mtrices with modulr nd sptil structure for neurl networks. Physicl Review E 9(4): Neftci E. Bins J. Rutishuser U. Chicc E. Indiveri G. nd Dougls R. J. (3). Synthesizing cognition in neuromorphic electronic systems. Proceedings of the Ntionl Acdemy of Sciences (37):E3468 E Neftci E. Chicc E. Indiveri G. nd Dougls R. (). A systemtic method for configuring vlsi networks of spiking neurons. Neurl computtion 3(): Peters A. nd Pyne B. R. (993). Numericl reltionships etween geniculocorticl fferents nd pyrmidl cell modules in ct primry visul cortex. Cererl Cortex 3: Rdford A. Metz L. nd Chintl S. (5). Unsupervised representtion lerning with deep convolutionl genertive dversril networks. CoRR s/ Rjn K. nd Aott L. F. (6). Eigenvlue spectr of rndom mtrices for neurl networks. Physicl Review Letters 97(884): Rumelhrt D. E. Hinton G. E. nd Willims R. J. (986). Lerning representtions y ck-propgting errors. Nture 33(688): Rutishuser U. nd Dougls R. J. (9). Stte-dependent computtion using coupled recurrent networks. Neurl Compututtion (): Shriki O. Hnsel D. nd Sompolinsky H. (3). Rte models for conductnce-sed corticl neuronl networks. Neurl Computtion 5(8): Somers D. C. Nelson S. B. nd Sur M. (995). An emergent model of orienttion selectivity in ct visul corticl simple cells. Journl of Neuroscience 5(8): Sussillo D. nd Aott L. F. (9). Generting coherent ptterns of ctivity from chotic neurl networks. Neuron 63(4): Theno Development Tem (6). Theno: A Python frmework for fst computtion of mthemticl expressions. rxiv e- prints s/ Tsumoto T. Eckrt W. nd Creutzfeldt O. D. (979). Modifiction of orienttion sensitivity of ct visul cortex neurons y removl of g-medited inhiition. Experimentl Brin Reserch 34(): Wei Y. (). Eigenvlue spectr of symmetric rndom mtrices for multicomponent neurl networks. Physicl Review E 85(666): Yoshimur Y. Dntzker J. L. M. nd Cllwy E. M. (5). Excittory corticl neurons form fine-scle functionl networks. Nture 433: