Bilateral semantic processing: Inferences in language, insight in problem solving. Mark Jung-Beeman

Transcription

1 Bilateral semantic processing: Inferences in language, insight in problem solving Mark Jung-Beeman Northwestern University Department of Psychology Neuroscience Institute Cognitive Brain Mapping Group \

2 Bilateral semantic processing: Inferences in language, insight in problem solving Northwestern University Zoe Clancy Jason Haberman (UCDavis) Sandra Virtue (Depaul U) Stella Arambel (deceased) Dianne Patterson Todd Parrish Paul Reber Terri Swan Karuna Subramaniam Ed Bowden Drexel University John Kounios Debbie Green Jennifer Frymiare (U Wisc) Jessica Fleck Richard Greenblatt Bar-Ilan University Miriam Faust Nira Mashal Research sponsored by NIDCD/NIH

3 Bilateral semantic processing: Inferences in language, insight in problem solving OUTLINE: Drawing inferences from stories -- bilateral comprehension Three bilateral component semantic processes (to start) Insight -- bilateral, parallel processing during problem solvin

4 Bilateral Activation, Integration, and Selection model of semantic processing Semantic activation - Wernicke s area Bottom-up lexical-semantic activation: index of semantic representations (pmtg) Semantic integration - anterior Sup. Temp. Gyr Compute semantic overlap - detect or generate (astg) Semantic selection - Inf. Frontal Gyrus Select among competing activated concepts (IFG)

5

6 Bilateral semantic processing: Inferences in language, insight in problem solving OUTLINE: Drawing inferences from stories -- bilateral comprehension Three bilateral component semantic processes (to start) Insight -- bilateral, parallel processing during problem solving

7 Problems with view that language is purely a LH functi General anatomical symmetry RH damaged patients - some language problems Recovery from aphasia, hemispherectomy, callosotomy Neuroimaging - always some RH signal, some tasks RH>LH Some tasks lvf-rh better than rvf-lh

8 Brain bases of comprehension of natural language Natural language, stories, discourse Higher level semantic processing (plus all lower levels) As language input more complex (and natural): More anterior temporal lobes More bilateral processing

9 Brain bases of cognitive processes when people draw inferences from stories Causal bridging (coherence) inferences Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.

10 Brain bases of cognitive processes when people draw inferences from stories Causal bridging (coherence) inferences Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes. He came out wearing his tuxedo, which had belonged to John' father, but looked like new.

11 Brain bases of cognitive processes when people draw inferences from stories Causal bridging (coherence) inferences Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes. He came out wearing his tuxedo, which had belonged to John' father, but looked like new. CHANGE

12 Brain bases of cognitive processes when people draw inferences from stories We know people make such causal inferences We know a lot about other types of inferences that peo make - types of text, motivation, knowledge, capacity We still don t know much about component processes that support this seemingly complex behavior

13 RH semantic processing and inferences RHD patients have difficulty drawing inferences Answer questions about inferable events less accurately than control subjects; intact on explicitly stated facts (Brownell et al., 1986; Beeman, 19 Do not show inference-related priming; control subjects do (Beeman, 19

14 Proposed component processes of inference generation 1) Activation / integration (detect overlap) 2) Selection 3) Incorporation / integration (map overlap) Hemispheric cooperation RH activates information that may support inferences. Weak activation not reach consciousness.

15 Time course of inference related semantic activation in both hemispheres during story comprehension. Before going to the wedding, John was sitting around in his jeans, 1 so he went to his bedroom to find some clothes. 2 After a few minutes, 3 he came out wearing his tuxedo, 4 which had belonged to John's father 5, but was still fashionable and looked like new. (1) and (2): Predictive inference. (3): Transition. (4): Coherence or bridging inference. (5): Resolved and incorporated. - CHANGE

16 Left visual field Right visual field Left Hemisphere Right Hemisphere

17 Before going to the wedding, John was sitting around in his jeans, 1 so he went to his bedroom to find some clothes. 2 After a few minutes, 3 he came wearing his tuxedo, 4 which had belonged to John's father 5, but was still fashionable and looked like new. Brain and Language, 2000 Priming: Inference faster than Unrel Test point rvf-lh lvf-rh

18 Asymmetric dynamic semantic fields: relatively coarser coding in RH; better selection in LH Left Hemisphere Right Hemisphere RULER foot TOES foot Small but strongly activated; Focused on dominant or contextually relevant concepts - easy to select, interpret, output CUT Large but weakly activate Diffuse, including second and less relevant concepts - hard to select, output

19 RH coarse semantic coding: Increased likelihood of semantic overlap for distant semantic relations foot foot glass pain glass pain

20 Bilateral Activation, Integration, and Selection model of semantic processing Semantic activation - Wernicke s area Bottom-up lexical-semantic activation: index of semantic representations (pmtg) Semantic integration - anterior Sup. Temp. Gyr Compute semantic overlap - detect or generate (astg) Semantic selection - Inf. Frontal Gyrus Select among competing activated concepts (IFG)

21 RH Middle & superior temporal gyrus involved in computing semantic integration Deriving theme from paragraphs Generating best ending (St. George et al (Kirchner et al.) Generating inferences? - moderately related sentence pairs (Mason & Just) Metaphoric over literal sentences (Bottini et al.) Detecting temporal/emotional inconsistency (Ferstl) Generating insight solutions (Jung-Beeman et al; Kounios et a

22 Brain activity when people draw inferences on-line, as indexed by fmri Three ways to contrast inference versus no-inference conditions: - Text: infernce versus no-inference; strong vs. weak constraint - Individual differences: high versus low Working Memory - Behavioral measures: recall of inferences General Results: Bilateral activity in pmtg; astg; IFG - modulated by constraint, WM, time

23 Brain activity when people draw inferences Inference: on-line, as indexed by fmri John was going to a wedding, but he had been sitting around the house in his jeans, so he went to his bedroom to find som clothes. Soon he came out wearing his tuxedo, * Explicit: went to his bedroom to change his clothes. Soon he came wearing his tuxedo,* - High baseline, ongoing stories; small input differen

24 Semantic integration at moment of implied events: Predominantly RH astg

25 Semantic integration at event point: Bilateral anterior Superior Temporal Gyrus L R Post Ant L R L R Lower (ns) threshold, selected for LH STG

26 Semantic activation and integration at coherence break ( tuxedo ): Predominantly LH STG

27 Semantic selection: High versus low working memory High WM (reading span) subs show stronger, earlier evidence of semantic selection of inferences (St. George et al; many behavioral Completion requires selection, incorporation

28 Semantic selection: Inferior frontal gyrus Selecting some concepts over competitors Usually IFG in LH (Thompson-Schill et al; Barch; Friston) Some instances, RH IFG (Seger 2000; Friederici et al., 2000; Jung-Beeman et al.

29 Semantic selection: Inferior frontal gyrus Selecting some concepts over competitors Usually IFG in LH (Thompson-Schill et al; Barch; Friston) Some instances, RH IFG Unusual verb generation (cake -> decorate ) (Seger 2000) Repair grammatical errors (Friederici et al., 2000) Utilize unintended meaning of ambiguous words in sentence (Jung-Beeman et al.

30 Semantic selection: fmri signal in IFG (LH > RH) at coherence break in High WM subs only (Fig: High WM > Low WM)

31 Replication and extension: Working memory and predictability Unpredictable inferences: LH activation, IFG, pstg searching for connections Predictable inferences: Bilateral activation, IFG, pstg building on connections Higher WM (n=13) > lower WM (n=13): building on connections facile comprehension

32 Successful integration versus continued activation: STG in High vs. Low WM subs at coherence break, Predictable inferences RH IFG High WM subs show bilateral (stronger in RH) Low WM show LH only R ps p<

33 Replication and extension: Working memory and predictability Unpredictable inferences: LH activation, IFG, pstg searching for connections Predictable inferences: Bilateral activation, IFG, pstg building on connections Higher WM (n=13) > lower WM (n=13): RH activation, pstg, IFG, and a little astg building on connections facile comprehension

34 Successful integration versus continued activation: STG in High vs. Low WM subs at coherence break, Predictable inferences RH astg High WM subs show bilateral (stronger in RH) p<.005 Low WM show LH only, no astg

35 Replication and extension: Working memory and predictability Unpredictable inferences: LH activation, IFG, pstg searching for connections Predictable inferences: Bilateral activation, IFG, pstg building on connections Higher WM (n=13) > lower WM (n=13): RH activation, pstg, IFG, an little astg building on connections facile comprehension

36 Conclusions about inferences Semantic integration builds up as story hints that some event might occur: anterior STG; RH (?) At coherence break: integration and activation (STG), especially in LH completing the inference requires selection (IFG) RH contributes to facile inferencing/comprehension not just kick in when demands are high

37 Current projects, Future directions Shift semantic distance for integration --> shift hemi asymmetry Closely tie to behavioral markers of inference activation, selection, incorporation Recall of inferences! Priming of inferences Successful integration versus effort of difficult integration Incorporation (recall study)

38 Recalled inferences If inferences recalled, must have been incorporated Working Memory correlates with total recall Recall of inferences NOT with recall of episodes w/o inferences Contrast fmri signal of recalled infs versus recall episode, no infs

39 Inferences recalled versus Episode recalled, inf not recalled L R Post Ant L R L R R R Bilateral pmtg, stronger in RH RH asts, bilat IFG p<.005, positive only

40 So what? Knowing where processing occurs informs and constrains what and how it occurs

41 Bilateral semantic processing: Inferences in language, insight in problem solving OUTLINE: Drawing inferences from stories -- bilateral comprehension Three bilateral component semantic processes (to start) Insight -- bilateral, parallel processing during problem solvin

42 Bilateral Activation, Integration, and Selection model of semantic processing Semantic activation - Wernicke s area Bottom-up lexical-semantic activation: index of semantic representations (pmtg) Semantic integration - anterior Sup. Temp. Gyr Compute semantic overlap - detect or generate (astg) Semantic selection - Inf. Frontal Gyrus Select among competing activated concepts (IFG)

43 Why does the RH code more coarsely Asymmetries in neural microcircuitry

44

45 Given topographic mapping of brain, broader input/output fields => coarser semantic codin Left Hemisphere Right Hemisphere RULER foot TOES foot CUT Small but strongly activated; Focused on dominant or contextually relevant concepts Large but weakly activate Diffuse, including second and less relevant concept

46

47 RH coarse semantic coding: Increased likelihood of semantic overlap for distant semantic relations foot foot glass pain glass pain

48

49 Why a separate area for semantic integration? Could form associations in activation area BUT Higher level relations, correlated co-occurrence, indir Ability to extract, attend to, & manipulate relations Analogous to individual areas within vision (e.g., motion)

50 Why anterior STS/STG for semantic integration? Again, neural architecture

51 Patchy organization and multisensory integration (Beauchamp 2004) L R Post Ant L R

52

53 Why anterior STS/STG for semantic integration? Again, neural architecture More anterior = longer intrinsic conxns, better t integrate across patches RH = longer than LH

54

55 Important clarifications Not an inference area Semantic integration - participates in many functions Not specific to categories of inferences - varies with demand Tight comparison not reveal whole network Just areas that differ when storied imply versus explicitly state events RH and LH cooperate

56

57

58 Bilateral semantic processing: Inferences in language, insight in problem solving OUTLINE: Drawing inferences from stories -- bilateral comprehension Three bilateral component semantic processes (to start) Insight -- bilateral, parallel processing in problem solving

59 Brain bases of insight during problem solving: Aha! and antecedents Most problems solved with mix of analytic and insight processing Distinct computations, distributed across hemispheres, allow two approaches to proceed simultaneously (partially interactive) Hemispheric components, task shielding/switching

60 Archimedes and the crown King s crown - gold, or silver Archimedes knew gold and silver differed in density Archimedes knew weight, but couldn t geometrically measure to obtain volume (and compute density)

61

62 Archimedes and the crown Why has story persisted so long?

63 Archimedes and the crown Why has story persisted so long? Resonates with our own experiences of solving insight problems solving problems with insight

64 Archimedes and the crown Solvers reach impasse (dead-end) - couldn t measure Must reinterpret some aspect of problem Volume by water displacement Unconscious processing important If not thinking of crown, how recognize importance of water? Solution accompanied by Eureka!

65 Insight component processes? Insight solutions associated with Switching to new strategy or associations ( restructuring ) Semantic integration -- solvers see connections that previously eluded them Right hemisphere?

66 Solving problems with insight Characteristics of both insight problems and solving processes similar to characteristics of discourse and comprehension processes for which the Right Hemisphere (RH) seems to make contributions Drawing inferences, understanding the gist Getting jokes, metaphors, connotations 2ndary word meanings

67 Solving problems with insight Solvers reach impasse (dead-end) Must reinterpret some aspect of problem Unconscious processing important Solution accompanied by Aha!

68 Short insight problems: Remote Associates Test: The RAT (Mednick, 1962 RAT Compound Remote Associate Proble Bowden & Jung Beeman, tennis child same strike lame scan

69 RAT Compound Remote Associate Proble Bowden & Jung Beeman, tennis child same strike lame scan

70

71 Aha! experience Solution appears sudden and obvious As soon as you think of solution, you just know it works for all three words Comes as a whole, not part by part (vs strategic, step-by-step testing, etc)

72 Event-related fmri design Insight solutions versus noninsight solutions Very tight comparison Not reveal whole network of problem solving Highlights just components that are uniquely engaged (o at least emphasized) for insight solutions

73 Insight effect in RH anterior Superior Temporal Gyrus: FMRI signal for insight > noninsight solutions. L R Post Ant L R L coronal R axial sagittal p <.005, cluster > 500 mm 3

74 RH astg: Singal change across the active region Signal change for insight Insight effect and noninsight solutions (Ins - non) Percent signal change Percent Signal change Time (sec)

75 Signal change for insight and noninsight solutions, in astg across hemispheres Percent signal change Insight Non Best clus within each hemisphere LH RH Hemisphere

76 Parallel study with 128 channel EEG Temporal specificity Processing specificity - frequencies

77 Gamma band insight effects

78

79

80

81 Insight solving conclusions Insight solutions associated with increased activity in RH astg Binding and conscious accessibility (gamma) over RH astg Preceded by visual gating (alpha) - RH temp/ occipital areas

82 Insight solving conclusions Insight solutions associated with increased activity in RH astg Binding and conscious accessibility (gamma) over RH astg - Lexical or semantic integration Preceded by visual gating (alpha) - RH temp/ occipital areas - Sensory gating indicates cognitive control?

83 Replication plus more areas New data set: improved N, scanner, protocol RH astg (distant semantic integration) Anterior Cingulate (monitoring response competition, switching) Posterior Cingulate - same? Hippocampus/parahippocampal gyri - memory, reorgnzn?

84 Insight effect in RH Superior Temporal Gyrus: FMRI signal for insight > noninsight solutions. L R Post Ant L R L coronal R axial sagittal p <.001, cluster > 1000 mm 3 ant and post STG

85 NONinsight effect in LH Inf. Frontal Gyrus: FMRI signal for NONinsight > insight solutions. LH IFG - dominant semantic retrieval or selection - turns on at problem onset - off at solution, esp y Insight RH IFG - unusual retrieval / selection - off at problem onset - on at solution (I>NI, ns) sagittal p <.005, cluster > 1000 mm

86 General vs specific mechanisms - Visual Aha!

87 Visual Aha! effect in RH anterior Mid Temporal Gyrus FMRI signal for insight > noninsight recognition L R Post Ant L R L coronal R axial sagittal p <.01, cluster > 500 mm 3

88 Visual Aha! effect in RH anterior Mid Temporal Gyrus FMRI signal for insight > noninsight recognition L R Post Ant L R L coronal R axial sagittal p <.01, cluster > 500 mm 3

89 Visual Aha! effect in RH Angular Gyrus: FMRI signal for insight > noninsight recognition L R Post Ant L R L coronal R axial sagittal p <.01, cluster > 500 mm 3 Also: RH Sup Frontal Gyru

90 Visual Aha! effect in Bilateral M. Occipital Gyri: FMRI signal for NONinsight > insight recognition L R Post Ant L R L coronal R axial sagittal p <.005, cluster > 500 mm 3

91 Visual Aha! conclusions NOT just for verbal problems Similarities - shared mechanisms (not insight, but ) Insight: top-down, cognitive control, integration RH -- unconscious, weak but mutually constraining, integration Recognition comes as a whole, not part by part Noninsight: bottom-up Some differences - Angular Gyrus somewhat surprising

92 General vs specific mechanisms - Visual Aha!

93 Insight solving conclusions Insight solutions associated with increased activity in RH astg Binding and conscious accessibility (gamma) over RH astg Preceded by visual gating (alpha) - RH temp/ occipital areas

94 Insight solving conclusions Insight solutions associated with increased activity in RH astg Binding and conscious accessibility (gamma) over RH astg - Lexical or semantic integration Preceded by visual gating (alpha) - RH temp/ occipital areas - Sensory gating indicates cognitive control?

95 Insight solving conclusions Insight solutions associated with Semantic integration -- solvers see connections that previously eluded them When the light goes on

96 Bilateral Activation, Integration, and Selection model of semantic processing Semantic activation - Wernicke s area Bottom-up lexical-semantic activation: index of semantic representations (pmtg) Semantic integration - anterior Sup. Temp. Gyr Compute semantic overlap - detect or generate (astg) Semantic selection - Inf. Frontal Gyrus Select among competing activated concepts (IFG)

97 Insight preparation Do different mental states influence how you solve problems? Brain activity during a rest period (fmri) or at a Ready? prompt (EEG), prior to getting a problem Problems solved with insight versus without insight

98 Preparation for Insight Is there a general form of preparation for insight that begins before a problem is presented? We examined neural activity during the 2 sec immediately before each problem was presented. Compared neural activity preceding problems solved with insight to activity preceding problems solved without insight.

99 8-9 Hz 9-10 Hz I - T t N - T +3.25

100

101

102 Conclusions Two forms of preparation. Noninsight: Increased visual attention to displayed problem. Insight: Mobilization and control of cognitive resources activation of temporal lobe semantic regions; suppression of irrelevant thoughts.

103 Summary Insight is different from ordinary problem solving. Insight involves a sudden, discrete, awareness of the solution to a problem. Insight involves different neural structures and mechanisms. Insight is the result of a special form of preparation involving cognitive regulation by medial frontal region.

104 Is insight really sudden? Part II: Antecedents of insight Positive mood facilitates insight and creative problem solving (Isen et al.)

105 Insight and mood Positive mood associated with increased creativity Better access to more distant associations Increased cognitive flexibility Anxiety associated with decreased creativity narrower focus of attention

106 Positive mood and insight Mood Positive mood enhances (anxiety impedes):» Total solution rate» % solved with insight» Insight-like preparatory activity in ACC

107 Positive mood modulates prep activity in ACC Insight >N Prep activi Pos Aff>N in prep ac Convergen

108

109 General vs specific mechanisms - Visual Aha!

110 Thank you!