Gabriel Kreiman Children s Hospital, Harvard Medical School

Size: px

Start display at page:

Download "Gabriel Kreiman Children s Hospital, Harvard Medical School"

Gerard Casey
5 years ago
Views:

1 Neural computa-ons: lessons from peeking inside the brain Gabriel Kreiman Children s Hospital, Harvard Medical School gabriel.kreiman@tch.harvard.edu hcp://klab.tch.harvard.edu

2 What computers can do [2014]

3 Humans vs. machines, 2014

Biologically-inspired computation :&%3*0'' ' E.,"$"/-.

4"/'3.'."#$).%&'(3$(#3,/')>"$'*3&&3)./')1'G"%$/')1'">)&#-).

High-resolution, highspeed sensors (e.g. cameras) Tools,

neuronal circuits Decoding activity in real time Writing-in

4 Biologically-inspired computation :&%3*0'' ' E.,"$"/-.4'/)&#-)./',)'53F(#&,'+$)7&"*/'6%>"'"*"$4"5',6$)#46'(6%.4"/'3.'."#$).%&'(3$(#3,/')>"$'*3&&3)./')1'G"%$/')1'">)&#-).' Massive computational power Wireless data transmission High-resolution, highspeed sensors (e.g. cameras) Tools, models, hardware Algorithms, solutions Listening to neuronal circuits Decoding activity in real time Writing-in information! the great events of the world take place in the brain (Oscar Wilde)

7 A flower, as seen by a computer

8 Vision as a summer project

9 Why is vision difficult?

10 Partial Summary!"! #$%&'()*$%+$,-$&.'*/-0+'0.+)(-01%&(-!-2+1/1,+0*//34+$(5+'&%- */,1'+)67(-.$%&'/3+$,-+$)&//+,&$)-0175.)*81$(-

11 Visual system circuitry Primary visual cortex (V1) Lateral geniculate nucleus Optic nerve Right visual field Left visual field Modified from Society for Neuroscience Brain Facts

12 Magic in the brain: ventral visual cortex NOTE: This is only a coarse description of the circuit Many (most?) connections are still probably missing We do not understand the functional role for most of the connections Felleman and Van Essen. Cerebral Cortex 1991

13 Neocortical circuits can be quite specific - Unable to visually recognize friends, famous people, relatives, even self - Could not learn to recognize new faces (but could learn to recognize new people from voice and other cues) - Normal language, memory, learning, non-face object recognition - Many normal visual functions Distribution of lesion sites in cases of face agnosia Damasio et al. Face agnosia and the neural substrates of memory. Annual Review of Neuroscience (1990). 13:89-109

14 Vision is a constructive process

15 Methods to study the brain at different scales Kreiman. Physics of Life Reviews 2004

16 Reading out the biological source code

17 Neurons show sensitivity to special visual features Newsome et al (1989) Nature 341:52-54 Hubel and Wiesel (1959) J. Physiol. 148: Kuffler, S. (1953) J. Neurophys. 16: Desimone et al (1984) J. Neurosci. 4:

Invasive physiological recordings in the human brain Patients with pharmacologically intractable epilepsy Multiple electrodes implanted to localize seizure

parahippocampal gyrus) Patients stay in the hospital for about 7-10 days Cannot choose type of electrodes Cannot choose number of electrodes Cannot choose

18 Invasive physiological recordings in the human brain Patients with pharmacologically intractable epilepsy Multiple electrodes implanted to localize seizure focus Targets typically include the temporal lobe (inferior temporal cortex, fusiform gyrus), medial temporal lobe (hippocampus, entorhinal cortex, amygdala and parahippocampal gyrus) Patients stay in the hospital for about 7-10 days Cannot choose type of electrodes Cannot choose number of electrodes Cannot choose electrode location Limits on recording time Many other limitations Itzhak Fried (UCLA) Joseph Madsen (Harvard) Alex Golby (Brigham and Women) Stanley Anderson (J. Hopkins)

40 micron diameter, impedance ~ 1 MOhm!Action potentials, LFPs ]3*"'L/"(P'!Subdural (temporal cortex, frontal cortex)!

19 A panoply of different types of electrodes ^'Lµ^P'!Targets typically include the medial temporal (hippocampus, entorhinal cortex, amygdala and parahippocampal gyrus)!40 micron diameter, impedance ~ 1 MOhm!Action potentials, LFPs ]3*"'L/"(P'!Subdural (temporal cortex, frontal cortex)!low impedance macro contacts (<1 kohm)!high impedance microwires (~ 1MOhm)!Large coverage Itzhak Fried (UCLA), Joseph Madsen (Harvard), Alex Golby (Brigham and Women), Stanley Anderson (Hopkins)

20 Reliable, selective and rapid responses in human inferior temporal cortex animals chairs faces *!- *=- *9- *:- *>- vehicles fruits <%.&3.']%.4' Inferior temporal gyrus 8$)//='H"/3*)."='_)4),6"-/='J3(6*).5=' ]%.%2%='^)4"&/='J)&&/=':)..)$/='E,)='\"$$",''

$/"/'%$"'/,%7&"')>"$'*#&-+&"'5%G/' TY'6)#$/' ST'6)#$/' Ea\'b'3.,$%($%.$

21 @"&"(->"'$"/+)./"/'%$"'/,%7&"')>"$'*#&-+&"'5%G/' TY'6)#$/' ST'6)#$/' Ea\'b'3.,$%($%.3%&'c"&5'+),".-%&' (1A'])&3%/='H3:%$&)='_")+)&5' `%./%&'",'%&'SOMS'

22 Tolerance to scale and rotation changes Liu et al 2009 Right Medial Temporal Gyrus, Parahippocampal Part (Talairach: [32,- 34,- 14])

23 Responses are tolerant to small amounts of clutter Left Occipito-Temporal Fusiform Gyrus [-42,-44,-24] Agam et al 2010

24 Highly selective and tolerant responses in the human medial temporal lobe n=9 n=14 n=13 n=7 n=5 n=7 n=6 n=6 n=7 n=4 n=11 n=11 n=10 n=7 n= ms Quian Quiroga, Reddy, Kreiman, Koch, Fried. Nature 2005

25 Electrical stimulation can bias visual perception Afraz et al. MicrosBmulaBon of inferotemporal cortex influences face categorizabon. Nature (2006) 442:

26 Electrical stimulation in the human brain Penfield & Perot. The brain's record of auditory and visual experience. A final summary and discussion. Brain (1963) 86:

27 Partial Summary MA! i.5"$/,%.53.4'."#$%&'(3$(#3,/'()5"/'!'`3)&)43(%&&gr3./+3$"5' %&4)$3,6*/'#.5"$&G3.4'3.,"&&34".,'()*+#,%-)./' 44-F1/&'*$0&-)1-)'*$(G1'7*81$(-H(0*/&I-51(+81$I-(17&-'1)*81$J- 44-?*5+%-'&(51$(&(-H!KK4!>K-7(J- -

28 Deciphering the neural code Hung et al, 2005

29 Machine learning approach to decode neural signals

31 Decoding selective and transformation tolerant information

33 Partial Summary MA! i.5"$/,%.53.4'."#$%&'(3$(#3,/'()5"/'!'`3)&)43(%&&gr3./+3$"5' %&4)$3,6*/'#.5"$&G3.4'3.,"&&34".,'()*+#,%-)./' SA! J"/+)./"/'%&).4',6"'>".,$%&'>3/#%&'/,$"%*'' RR'E.($"%/"'3.'$"("+->"'c"&5'/3g"/' RR'])&"$%.("',)',$%./1)$*%-)./'L/(%&"='+)/3-).='/)*"'$),%-).P' RR'J%+35'$"/+)./"/'LMOORMUO'*/P' 9"! L&-0*$-.(&-*-7*06+$&-/&*'$+$,-*55'1*06-)1-'&*%-1.)-M+1/1,+0*/- 01%&(-+$-(+$,/&-)'+*/(- -

34 From biological code to computer code Felleman and Van Essen. Cerebral Cortex 1991 Fukushima, Rolls, LeCun, Wallis, Mel, Riesenhuber, Poggio Riesenhuber&Poggio 1999; Serre et al 2007

35 A biologically-inspired, bottom-up, hierarchical model of object recognition Cadieu, Knoblich, Kouh, Mutch, Riesenhuber, Serre, Poggio

Scale and position tolerance when decoding from ITC and model units Support vector machine classifier Linear kernel Pseudo- popula-on of 64 inferior temporal

37 Scale and position tolerance when decoding from ITC and model units Support vector machine classifier Linear kernel Pseudo- popula-on of 64 inferior temporal cortex neurons [white] Model: 64 random C2- level units Categoriza-on performance Chance = 1/8 Cross- valida-on with Chou Hung, Jim DiCarlo, Tommy Poggio Hung et al 2005

38 Model performance in the presence of clutter

39 Towards understanding vision in real scenes

Partial Summary MA! i.5"$/,%.53.4'."#$%&'(3$(#3,/'()5"/'!'`3)&)43(%&&gr3./+3$"5' %&4)$3,6*/'#.

($"%/"'3.'$"("+->"'c"&5'/3g"/' RR'@"&"(->3,G',)'53j"$".,'/6%+"/' RR'])&"$%.("',)',$%./1)$*%-).

"'&"%$.3.4'%++$)%(6',)'$"%5')#,'73)&)43(%&' ()5"/'3.'/3.4&"',$3%&/' :"! N+@+%&-*$%-01$O.

40 Partial Summary MA! i.5"$/,%.53.4'."#$%&'(3$(#3,/'()5"/'!'`3)&)43(%&&gr3./+3$"5' %&4)$3,6*/'#.5"$&G3.4'3.,"&&34".,'()*+#,%-)./' SA! J"/+)./"/'%&).4',6"'>".,$%&'>3/#%&'/,$"%*'' RR'E.($"%/"'3.'$"("+->"'c"&5'/3g"/' RR'])&"$%.("',)',$%./1)$*%-)./'L/(%&"='+)/3-).='/)*"'$),%-).P' RR'J%+35'$"/+)./"/'LMOORMUO'*/P' WA! Z"'(%.'#/"'%'*%(63."'&"%$.3.4'%++$)%(6',)'$"%5')#,'73)&)43(%&' ()5"/'3.'/3.4&"',$3%&/' :"! 71%&/-0*$-0*5).'&-&((&$8*/(-*(5&0)(-1G-1MR&0)-'&01,$+81$- -

41 Objects can be recognized from partial information

42 Object completion task

43 Performance in object completion task SO'7#77&"/' MO'7#77&"/' Y'7#77&"/' T'7#77&"/'

Limited object completion in feed-forward model SOOO'n:So'#.3,/'3.',6"'*)5"&'?)5"&'$"/+)./"/',)'SU'"m"*+&%$')7p"(,/' :)./35"$'SO'#.3,/'K3,6'6346'@!J'L,$%3.3.4'5%,%P' UOO'$"+"--).

44 Limited object completion in feed-forward model SOOO'n:So'#.3,/'3.',6"'*)5"&'?)5"&'$"/+)./"/',)'SU'"m"*+&%$')7p"(,/' UOO'$"+"--)./'K3,6'53j"$".,'7#77&"'&)(%-)./' ]$%3.'(&%//3c"$'K3,6'XOq')1',6"'$"+"--)./' ]"/,'(&%//3c"$').'$"*%3.3.4'WOq')1',6"'$"+"--)./' E5".-c(%-).',%/2'L(6%.("bTqP' L1#&&'3*%4"/P' &'3*%4"/P'

45 Example responses during object completion Inferior Temporal Gyrus

46 Delayed responses to partial objects

Object completion requires more time (behavior)!r'?

150 SOA 20 0 50 100 150 SOA 20 0 50 100 150 SOA?

47 Object completion requires more time 100 Full 100 Occluded 100 Occ Scrambled Performance SOA SOA SOA?I@9' 100 Full, Mask 100 Occluded, Mask 100 Occ Scrambled, Mask Performance SOA SOA SOA Z%&,"$'<%$5"/,G='<%.&3.']%.4'

48 Top-down / recurrent signals may ameliorate the problem of missing information IC$%(,)$'.",K)$2/'(%.'/)&>"',6"'+$)7&"*')1'+%C"$.'()*+&"-).'L"A4A'<)+c"&5P'' C 2 z(t) = C 2 z(t! 1) + # p = prototypes (fixed) i = 1,!,25!, n = parameters d = Euclidian distance i "(p i! C 2 z(t! 1)) d(z(t -1), p i ) n :%&3.'`#3%='<%.&3.']%.4'

49 Proof-of-principle: Adding top-down signals improves recognition performance under occlusion SOOO'n:So'#.3,/'3.',6"'*)5"&'?)5"&'$"/+)./"/',)'SU'"m"*+&%$')7p"(,/' UOO'$"+"--)./'K3,6'53j"$".,'7#77&"'&)(%-)./' ]$%3.'(&%//3c"$'K3,6'XOq')1',6"'$"+"--)./' ]"/,'(&%//3c"$').'$"*%3.3.4'WOq')1',6"'$"+"--)./' E5".-c(%-).',%/2'L(6%.("bTqP'

50 Top-down connections help perform object completion!b!#*7"$' )1'7#77&"/' :&%//3c(%-).'+"$1)$*%.("' H"%.'ZG%C"='J%.5%&&'r;J"3&&G='<%.&3.']%.4' ]3*"'L*)5"&'(G(&"/P'

51 Summary (Object completion) "!SMR&0)-0175/&81$-5'&(&$)(-*-06*//&$,&-G1'- 5.'&/3-M1Q174.5-*'06+)&0).'&(- - "!T&.'*/-(+,$*/(-+$-6+,6&'-@+(.*/-*'&*(-'&7*+$- (&/&08@&-%&(5+)&-(61U+$,-1$/3-*-(7*//- G'*081$-1G-*$-1MR&0)- "!SMR&0)-0175/&81$-'&O.+'&(-*%%+81$*/ )*81$-H+"&"-71'&-87&J-HM&6*@+1'*/-*$%- 563(+1/1,+0*/-&@+%&$0&J- "!?&0.''&$)-*$%V1'-)154%1U$-01$$&081$(- 0*$-+75'1@&-'&01,$+81$-1G-5*'8*/-1'- 100/.%&%-1MR&0)(' "!F154%1U$-(+,$*/(-0*$-&$6*$0&-'&01,$+81$-.$%&'-*-@*'+&)3-1G-'&/*)&%-'1/&(-H$1)-(61U$J' "!?#&-+&"'cm%-)./'5#$3.4',%$4",'/"%$(6',%/2/' "!:&#C"$"5'/("."/'

52 Acknowledgments Hanlin Tang Jed Singer Hesheng Liu Yigal Agam Joseph Madsen Stan Anderson Thomas Serre Tomaso Poggio Thomas Miconi Dean WyaCe

53 Recommended books

54 Neural computa-ons: lessons from peeking inside the brain Gabriel Kreiman Children s Hospital, Harvard Medical School gabriel.kreiman@tch.harvard.edu hcp://klab.tch.harvard.edu

55 No change in amplitude

56 Holis-c responses (?)

57 Increased latency differences in higher visual areas

58 No change in amplitude. Change in selec-vity.

59 Example responses during object comple-on Leu Fusiform Gyrus

60 No changes in eye movements during object comple-on

61 Matched amplitude and matched decoding comparisons Response amplitudes matched Decoding performance matched

62 Example responses in the gamma frequency band Hz Fusiform gyrus

63 Example responses during object comple-on animals chairs a1 a2 a3 a4 a5 faces vehicles fruits

64 Delayed responses to par-al objects

65 ]$%(23.4'"G"'+)/3-).'%.5'*%++3.4'$"("+->"'c"&5/' :%&37$%-).'-*"0'x'SO'/"(/' :%.',$%(2')."'"G"' <"%5'*)>"*".,',)&"$%.("' J"%&'-*"'1""57%(2'

IFP signals are localized within ~10 mm Units: < 200 µm LFPs: 0.

66 IFP signals are localized within ~10 mm Units: < 200 µm LFPs: mm IFPs: < 15 mm EEG: ~10 cm IFP = intracranial field poten-al

67 Z"'(%.'5"()5"')7p"(,'3.1)$*%-).'1$)*',6"'*)5"&'#.3,/' :6)#'<#.4='[3*'H3:%$&)='])**%/)'\)443)'

68 Eye posi-on was near the fixa-on point during the ini-al ~200 ms Note: 2 subjects only

69 Example neurophysiological responses [1] Subject m00026 Channel 49

70 Example neurophysiological responses [1 ] Subject m00026 Channel 49

71 Example neurophysiological responses [2] Subject m00032 Channel 21

72 OBJECT COMPLETION

s34%&'i4%*='<"/6".4'_3#='[)/"+6'?%5/".' @"&"(->3,G'3.

73 ]%&%3$%(60'dRTQAQ=RYNAM=RMMAQe'' :&%//3c(%-).'+"$1)$*%.("'b'YUyUq'L(6%.4"bUOqP'

74 Example: Reliable and selective responses to a movie

75 Action potentials versus field potentials Ac8on poten8als Field poten8als Neurons communicate via ac-on poten-als The biophysics underlying ac-on poten-als is rela-vely well understood Typically, ac-on poten-als show stronger specificity than field poten-als Ul-mately, our computa-onal models are inspired by and neurons and synapses. The models in turn make predic-ons about neurons and synapses We can examine areas currently not studied with ac-on poten-als in the human brain We can sample a large number of brain areas Spa-al scale of ~0.5 to 20 mm High signal- to- noise ra-o Strong temporal stability Comparable trial- to- trial variability to ac-on poten-als Biophysics less clearly understood

76 Reliable and selective responses to a movie s34%&'i4%*='<"/6".4'_3#'

Chapter 7: First steps into inferior temporal cortex

BEWARE: These are preliminary notes. In the future, they will become part of a textbook on Visual Object Recognition. Chapter 7: First steps into inferior temporal cortex Inferior temporal cortex (ITC)