Pupillometry in language acquisition research?

Transcription

1 Pupillometry in language acquisition research? Tom Fritzsche CLEAR workshop Groningen September 10, 2010

2 Abstract Pupil size is not only influenced by exogenous factors (e.g. luminance) but by endogenous ones as well, especially emotion, arousal and cognitive load. Automatic eyetrackers log pupil size along with gaze data - so why not make use this information? The technique of pupillometry/teprs is introduced and findings from the literature are presented before discussing methodological issues and the potential application in language acquisition research. 2

3 Outline 1. Why pupillometry 2. Basics 3. Classics Hess & Polt 1960, Kahneman & Beatty Infant studies Gredebäck & Melinder 2010, Jackson & Sirois (2009) 5. Pupil size & language Just & Carpenter (1993); Engelhardt et al. (2010) 6. Methodology 7. Example 8. Discussion 3

4 1. Why pupillometry? "Pupil workshop" at the University of Potsdam in March 2010 Increasing no. of papers in the last years 4

5 1. Why pupillometry? Pupil size is A sensitive, reliable, consistent measure of cognitive load A dynamic (online) measure of processing Ideal for infant research Provided by most automatic eyetrackers e.g. Tobii, SMI, EyeLink Compatible with existing VoE paradigms Available in addition to other measures 5

6 2. Basics: History Connection between cognitive processing and pupillary dilation at the end of the 19th century "Every active intellectual process, every psychological effort, every exertion of attention, every active mental image, regardless of content, particularly every affect just as truly produces pupil enlargement as does every sensory stimulus." Oswald Bumke in 1911, (cited from Hess 1975 p ) Pupillometry pioneer: Otto Lowenstein Thompson 2005 Irene E. Loewenfeld (1993) The Pupil: Anatomy, Physiology and Clinical Applications. Iowa State University Press. book comprises the work of Lowenstein and Loewenfeld standard reference on pupil research Since 1960s: Jackson Beatty, Eckhard Hess, Daniel Kahneman 6

7 2. Basics: Parameters Pupil size controlled by two opposing muscles Typical size: 3-5 mm(range: 1-9 mm) gets smaller by.04 per year Psychosensory reflex After any sensory occurence, external (visual, auditory, tactile etc.) or internal (emotions, mental processes, intentional effort), > dilation Exception: light -> constriction Hippus (pupillary unrest) Movements: Begin after 200 ms Peak.5 to 2 s after stimulus Pupil diameter covaries with Kahneman 1973, Bradley et al Skin conduction Heart rate (less reliably) from: Beatty & Lucero-Wagoner

8 2. Basics: Influencing factors Luminance Arousal Emotion Attention Cognitive load/intensity, resource allocation, mental effort Task complexity Intelligence Ahern & Beatty 1977, van der Meer et al

9 2. Basics: TEPR TEPR = task-evoked pupillary responses Non-reflexive phasic pupillary movement No causal link to but reporter variable for brain functions Changes of about.1-.5 mm Independent of baseline dilation Bradshaw 1969 Comparable between participants and different labs (Normalization unnecessary Beatty & Lucero-Wagoner 2000) Response delay ms Peak at about 1200 ms 9

10 2. Basics: TEPR index of cognitive load Arithmetic problems Hess & Polt 1964 Memory STM load Kahneman & Beatty 1966 LTM retrieval Beatty & Kahneman 1966 Perception (e.g. pitch discrimination) Language Syntactic complexity Schluroff 1982, Just & Carpenter 1993 Grammaticality violations Gutiérrez & Shapiro 2010 Sentence comprehension Wright & Kahneman 1971 Context integration Engelhardt et al Attention Beatty 1982, 1988 Responding (similar to ERPs - CNV or RP) Standard tests of "concentration" 10

11 3. Classics: Hess & Polt 1960 Pictures shown for 10 s after a 10 s baseline Participants: 6 (2, 4 ) Data points every 500 ms Highly reliable results (1 day retest) Interpretation: Pupil size reflects interest value Bidirectional TEPR (later: no confirmation) pleasant > dilation unpleasant > constriction 11

12 3. Classics: Kahneman & Beatty 1966 Pupil size reflects memory load Participants: 5 ( ) digit (3-7) memory task Memorization: Dilation Recall: Constriction Practice effects: reduction in repetition -> processing load 12

13 4. Infant Studies: Gredebäck & Melinder 2010 Action understanding in infants Participants 28 6-month olds month olds Pupil size varies with rationality Larger dilation for non-rational vs. rational goal Replication with familiarity bias removed 13

14 4. Infant Studies: Jackson & Sirois 2009 Event understanding: Violation of Expectiation (VoE) paradigm Participants: 24 8-month olds Pupil size bigger for impossible vs. possible novel events 14

15 Pupil size & 5. Just & Carpenter 1993 language: Reading study with 35 adults Processing in subject vs. object relative clauses The reporter that... attacked the senator the senator attacked... admitted the error publicly after the hearing. Measurement after first fixation of the main verb for max. 3 s Results Larger pupil size in object vs. subject RCs Later peak in ORC vs. SRC (116 ms) Pupil size is proportional to reading time error rate (of probe questions) TEPR reflects intensity of processing 15

16 Pupil size & 5. Engelhardt et al language: Listening study with 2x18 adults Integration of prosody and visual context in garden-path sentences While the woman cleaned (#) the dog that was big and brown stood in the yard. Measurement after 200 ms of onset disambiguating word for 1200 ms Results Larger pupil size for conflicting vs. congruent prosody If additional visual context was consistent (woman cleans not a dog): no effect of prosody inconsistent (woman cleans a dog): effect of prosody TEPR reflects processing effort of spoken language comprehension 16

17 6. Methodology: Luminance Pupillary light reflex (initial constriction) is much bigger and masks TEPRs CONTROL for - Luminance of stimuli - Lighting of room from: Bradley et al

18 6. Methodology: Steps in pupil analysis 1. Data inspection & artifact rejection 2. Noise filtering Low pass with 10 or 15 Hz 3. Replacement of missing data Use of other eye (if available) Linear interpolation 4. Averaging both pupil size values 5. Baseline creation 6. Calculating baseline-adjusted measures 7. (Normalizing) 18

19 6. Methodology: Pupillometry parameters Mean size Average in an interval of interest Peak size Maximum dilation in an interval of interest Peak latency Amount of time before peak size is reached Baseline Often 500 ms before critical stimulus Neutral stimulus 19

20 6. Methodology: Pupil size units Diameter units mm Tobii: mean/real (difference: T/X series vs. 1750) SMI: horizontal and vertical pixels (SMI) arbitrary unit (EyeLink) Pupil area (also different units) Not directly comparable between different systems Not comparable between different participants (in raw form - like EEG data) 20

21 6. Methodology: Normalization Really needed? Normalized data, relative to Baseline Participant's pupil mean Condition's pupil mean 21

22 6. Methodology: Analysis techniques Means (of time windows) Laeng et al. 2007; Falk-Ytter 2008 Slope in a time window Engelhardt et al B-splines Jackson & Sirois 2009 Like EEG/ERP data PCA Fixation-based pupil measures 22

23 6. Methodology: Pupil vs. Gaze Property Pupil Gaze Delay ms peak at 1200 ms 200 ms Repetition effects No Yes AoI-based analysis No Yes Light sensitivity High None Missing Data Easy to interpolate Hard to recover Design should be optimized for pupil or gaze 23

24 7. Ex: Visual Categorization Study Fritzsche & Höhle Can infants categorize different nonsense syllable strings by mapping them onto two characters? Familiarization phase with 2 conditions: FSG, PSG Participants 17 five-year-old children (mean 5.7 yrs) 24 adults (mean 23.5 yrs) 24

25 7. Ex: Stimuli Familiarization Trial Test Trial 25

26 7. Ex: Gaze data 26

27 7. Ex: Pupil data management 1. Linear interpolation of missing data data points chunk length 5yo 35% 29 adults 21% 8 2. Combining both pupil size values into one 3. Baseline 500 ms pre-stimulus 4. Normalization by mean participant pupil diameter (just to see what changes) 27

28 7. Ex: Pupil data raw baseline data data 28

29 7. Ex: Summary TEPR gives additional information Greater dilation in PSG vs. FSG condition for the familiarization in adults (although equal gaze patterns) This condition has more correct looks in the test phase TEPR seems to index the processing load of processing stimuli of this condition 29

30 So... Pupillometry in language acquisition research? It's worth a try! 30

31 8. Discussion 31

32 References Ahern, S. K. & Beatty, J. (1979). Pupillary responses during information processing vary with scholastic aptitude test scores. Science 205: Beatty, J. (1982). Phasic not tonic pupillary responses vary with aduitory vigilance performance. Psychophysiology 19(2): Beatty, J. (1988). Pupillometric signs of selective attention in man. In: G. C. Galbraith, M. L. Kietzman & E. Donchin (Eds.), Neurophysiology and Psychophysiology: Experimental and Clinical Applications. Hillsdaly, NJ: LEA: Beatty, J. & Kahneman, D. (1966). Pupillary changes in two memory tasks. Psychonomic Science 5: Beatty, J. & Lucero-Wagoner, B. (2000). The pupillary system. In: J. T. Cacioppo, L. G. Tassinary & G. G. Berntson (Eds.), Handbook of Psychophysiology 2nd ed. Cambridge, UK: Cambridge University Press: Bradley, M. M., Miccoli, L., Escrig, M. A. & Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4): Bradshaw, J. L. (1969). Background light intensity and the pupillary response in a reaction time task. Psychonomic Science 14: Engelhardt, P. E., Ferreira, F., & Patsenko, E. G. (2010). Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology 63(4): Falck-Ytter, T. (2008). Face inversion effects in autism: A combined looking time and pupillometric study. Autism Research 1(5): Gredebäck, G. & Melinder, A. (2010). Infants understanding of everyday social interactions: A dual process account. Cognition 114(2): Gutiérrez, R. S. & Shapiro, L. P. (2010). Measuring the time-course of sentence processing with pupillometry. CUNY Conference on human sentence processing. Hess, E. H. (1975). The Tell-Tale Eye: How Your Eyes Reveal Hidden Thoughts and Emotions. New York: Van Nostrand Reinhold. Hess, E. H. & Polt, J. M. (1960). Pupil size as related to interest value of visual stimuli. Science 132: Hess, E. H. & Polt, J. M. (1964). Pupil size in relation to mental activity during simple problem-solving. Science 140: Jackson, I. & Sirois, S. (2009). Infant cognition: Going full factorial with pupil dilation. Developmental Science 12(4): Just, M. A. & Carpenter, P. A. (1993). The intensity dimension of thought: Pupillometric indices of sentence processing. Canadian Journal of Experimental Psychology 47(2): Kahneman, D. (1973). Attention and Effort. Englewood Cliffs, NJ: Prentice-Hall. Kahneman, D. & Beatty, J. (1966). Pupil diameter and load on memory. Science 154: Klingner, J., Kumar, R. & Hanrahan, P. (2008). Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 Symposium on Eye Tracking Research and Applications: Laeng, B., Waterloo, K., Johnsen, S. H., Bakke, S. J., Låg, T., Simonsen, S. S., & Høgsæt, J. (2007). The eyes remember it: Oculography and pupillometry during recollection in three amnesic patients. Journal of Cognitive Neuroscience 19(11): Loewenfeld, I. E. (1993) The Pupil: Anatomy,Physiology and Clinical Applications. Iowa State University Press. Schluroff, M. (1982). Pupil responses to grammatical complexity of sentences. Brain and Language 17(1): Thompson, H. S. (2005). Otto Lowenstein, Pioneer Pupillographer. Journal of Neuro-Ophthalmology 25(1): van der Meer, E., Beyer, R., Horn, J., Foth, M., Bornemann, B., Ries, J., Kramer, J., Warmuth, E., Heekeren, H. R. & Wartenburger, I. (2010). Resource allocation and fluid intelligence: Insights from pupillometry. Psychophysiology 47(1): Wright, P. & Kahneman, D. (1971). Evidence of alternative strategies of sentence retention. Quarterly Journal of Experimental Psychology 23(2):