EMLAR VII EYE TRACKING: READING Arnout Koornneef a.w.koornneef@uu.nl OVERVIEW TUTORIAL Lecture (50 min) Basic facts about reading Examples Advantages and disadvantages of eye tracking Short break (10 min) Computer Assignments (30 min) Looking at some data Discussion/questions CLAIM: BEHAVIORAL METHODS the eyes can tell us a lot about the architecture of the cognitive system, including the architecture of the language system S R BLACK INFERENCES BOX (LINGUISTIC) THEORY THE EYE-TRACKER THE EYE-TRACKER 1
THE OLD EYE-TRACKER THE EYE-TRACKER monitors eye-movements from millisecond to millisecond provides information about where people look and for how long EXAMPLE SOME FACTS ABOUT READING people do not read a text smoothly, but fixate a particular word (200-300 msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew) HOW DO WE INTERPRET THE READING PATTERNS? READING TIME EXPERIMENTS THE LOGIC: Mental operations take time. If a mental operation is difficult for the language system, it will take more time. & If more operations (more steps) are needed, reading times will also be longer. 2
DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region EXPLANATION OF DIFFERENT MEASURES Bart annoyed Homer because 1 2 5 3 4 6 7 Reading Times for word 3 (Homer) First fixation duration = 3 First-pass duration = 3 + 4 Regression Path duration = 3 + 4 + 5 + 6 Second-pass duration = 6 Total duration = 3 + 4 + 6 DIFFERENT FIRST-PASS MEASURES FIRST FIXATION DURATION HOW DO WE INTERPRET THE READING TIMES? FIRST-PASS DURATION REGRESSION PATH DURATION TIME THE LINKING PROBLEM EYE-MIND ASSUMPTION (JUST & CARPENTER, 1980) eye mind Readers retain fixation on a word until processing is completed This includes processes like word recognition, syntactic parsing, semantic integration, referential integration 3
AN IDEAL WORLD (VAN BERKUM, 2004) THE REAL WORLD IS A REAL MESS (VAN BERKUM, 2004) Snowwhite kissed a dwarf Snowwhite kissed a dwarf TIME TIME W = word recognition; P = parsing; S = semantic integration; R = referential integration W = word recognition; P = parsing; S = semantic integration; R = referential integration THE REAL WORLD IS A REAL MESS In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc.) Thus, also in eye-tracking the effects of a manipulation are often visible more downstream (i.e. after the critical word or region). I call this cognitive spill-over LINKING ASSUMPTION (BOLAND, 2004) The eyes do not leave a word until it has been structurally integrated (tree building). Therefore, constraints that control structure-building influence first-pass reading time. Other measures (e.g., regression path duration) are sensitive for higher level processes (semantic integration, discourse processes) SOLUTION LINKING PROBLEM? Advantages and disadvantages of eye-tracking Perhaps the different measures can provide information about what is happening? (this is an empirical question) Eye tracking vs. self-paced reading Eye tracking vs EEG 4
Self-paced reading (the poor man s eye-tracker) Eye-tracking vs. self-paced reading Self-paced reading (the poor man s eye-tracker) Self-paced reading (the poor man s eye-tracker).............-... The..........-... Self-paced reading (the poor man s eye-tracker) Self-paced reading (the poor man s eye-tracker)... poor.......-......... man s...-... 5
Self-paced reading (the poor man s eye-tracker) Advantages eye tracking More natural (ecological validity) Higher temporal resolution.......... eye-tracker An example: Implicit causality and pronoun resolution What is an implicit causality verb? Bart annoyed Homer because he was making funny faces all day long! 6
Bart annoyed Homer because he just gets annoyed about a lot of things! Implicit Causality Bias Interpersonal verbs like annoy supply information about whose behaviour or state is the more likely immediate cause of the event at hand (= implicit causality bias) annoy has a bias towards subject (NP1-bias) praise has a bias towards object (NP2-bias) Marge praised Bart because he had behaved all day. Inconsistency-effect If the information explicitly provided in the remainder of the sentence goes against the bias of the verb at hand, people need more time to read the entire sentence (Caramazza et al., 1977). Consistent continuation (NP2-verb) Marge praised Bart because he had behaved all day. Inconsistent continuation (NP2-verb) Marge praised Bart because she likes to praise people. The issue Two accounts When during comprehension do people use this verb-based implicit causality information? Immediate Focus Account Integration Account 7
Immediate focus account Implicit causality information is used immediately Because of the implicit causality bias of the verb either the subject (NP1-bias) or object (NP2-bias) becomes more activated in the discourse model. Inconsistent pronoun is more difficult to resolve because it refers to a less activated discourse entity effect at the pronoun Clause-final integration account Implicit causality information is used late Implicit causality affects comprehension only at the end of the sentence when the main clause and subordinate clause are integrated into one representation Integration is more difficult if the information in the subordinate clause is inconsistent with the bias of the verb in the main clause effect after the pronoun Stimulus-example (translated from Dutch) Self-paced reading experiment David and Linda were both driving pretty fast. At a busy intersection they crashed hard into each other bias-consistent continuation David apologized to Linda because he according to the witnesses was the one to blame. bias-inconsistent continuation Linda apologized to David because he according to the witnesses was not the one to blame. Predictions The immediate focus account predicts an early effect at the inconsistent pronoun. 450 425 400 Reading Times consistent inconsistent The clause-final integration account predicts a later effect 375 350 325 w-4 w-3 w-2 w-1 w0 w1 w2 w3 w4 w5 con: Linda praised David because he had been able to complete inc: David praised Linda because he* had been able to complete 8
Conclusion Implicit causality information becomes available rapidly Evidence in favour of immediate focusing? Experiment 2 Eye-tracking study Same stories presented at once, thus, readers can regress But, not at the pronoun maybe integration? 450 400 350 300 250 First pass reading times consistent inconsistent Conclusion Eye tracking higher temporal resolution Probably because self-paced reading also suffers from task-induced spill-over 200 w-4 w-3 w-2 w-1 w0 w1 w2 w3 w4 w5 con: Linda praised David because he had been able to complete inc: David praised Linda because he* had been able to complete However..advantages self-paced reading No skipped words (i.e. a measurement for every word) Blinking is less problematic We will return to these issues Eye tracking vs. EEG 9
Disadvantage eye-tracking EEG research We know when something is happening, but we don t know what is happening. Perhaps EEG can help us (disadvantage EEG less natural) Moreover: You can only use skilled readers (no infants and language-disordered populations).this is possible in spoken language paradigms (Visual World Paradigm) EEG research Measuring electric activity of the brain at the scalp Different Event Related Potentials (ERPs) N400 is a negative shift 400 msec after stimulus onset (semantic processing) P600 is a positive shift 600 msec after stimulus onset (syntactic processing) Experiment 3 EEG-experiment Same stimuli presented visually Inconsistency effect and its scalp distribution he he cw+1 cw+2 The P600 Traditionally associated with syntactic processing (e.g., morphosyntactic violations) P600 10
Proposed model: Immediate anticipation Implicit causality is used to anticipate a plausible upcoming referent An bias-inconsistent pronoun is perceived as having the wrong syntactic gender Take home message. Be aware every method has its advantages and disadvantages Eye tracking High temporal resolution Natural reading process But No clear information about what is happening (i.e. you have to infer that) Problems and solutions Blinks Word skipping Spill-over How many items? Presentation of items Blinks There are several ways to deal with fixations containing blinks. Treat blinks like regular fixations. Remove only the blinks. Remove only the region that is affected by blinks Remove entire trial Word skipping If a participant skips a word, the first pass reading times (first fixation, first pass, regression path) will be zero Solution: Treat reading times of 0 as missing data (check whether the distibution is the same in your conditions!) Expand your regions Make sure the words in your critical region are relatively long Spill-over Keep at least 4-5 words constant after the critical region/word bias-consistent continuation David apologized to Linda because he according to the witnesses was the one to blame. bias-inconsistent continuation Linda apologized to David because he according to the witnesses was not the one to blame. 11
How many items? As a rule-of-thumb 10 measurements(trials) per condition Between subject design, 2 conditions means 10 trials for each participant Within subject design (e.g. latin square) means 20 trails (10 in each condition this can add up!) How many participants? Depends on what you are testing and your design (statistical issue ) As a rule-of-thumb In a standard within subjects design around 40 Presentation To reduce noise, make sure the critical region is not at the end nor at the beginning of a line David apologized to Linda because he according to the witnesses was the one to blame. Presentation To reduce noise, make sure the critical region is not at the end nor at the beinning of a line David apologized to Linda because he according to the witnesses was the one to blame. Presentation To reduce noise, make sure the critical region is not at the end nor at the beinning of a line After the break Looking at some data David apologized to Linda because he according to the witnesses was the one to blame. 12
Experiment Implicit causality (again ) How important is the use of the connective because? Three different connectives: because (want) but (maar) and (en Examples (approximate translation) David and Linda were both driving pretty fast. At a busy intersection they crashed hard into each other Consistent pronoun, because David apologized to Linda because he according to the witnesses was the one to blame. Inconsistent pronoun, because Linda apologized to David because he according to the witnesses was not the one to blame. Consistent pronoun, but David apologized to Linda but he according to the witnesses was not the one to blame. Inconsistent pronoun, but Linda apologized to David but he according to the witnesses was the one to blame. Consistent pronoun, en David apologized to Linda and he according to the witnesses was very confused due to the situation. Inconsistent pronoun, en Linda apologized to David and he according to the witnesses was fortunately able to except her apologies Results?? First-Gaze (because) 350 consistent inconsistent 300 250 200 2 3 4 because he had been able to Results Inconsistency effect for connective because No effects for but and and Implicit causality effects critically depends on a backward causal connective (i.e. because, omdat, want) 13