Page # Perception and Action. Lecture 3: Perception & Action. ACT-R Cognitive Architecture. Another (older) view

Transcription

1 Perception and Action Lecture 3: Perception & Action Cognition is a lot, but not enough! We want models to perceive stimuli in the outside world respond by acting in the outside world This gives us a more much faithful model! why? Today we ll look at Perceptual-Motor in ACT-R use to be called ACT-R/PM now integrated CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 1 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 2 ACT-R Cognitive Architecture Another (older) view Intentional Module (not identified) Declarative Module (Temporal/Hippocampus) Goal Buffer (DLPFC) Retrieval Buffer (VLPFC) Productions (Basal Ganglia) Matching (Striatum) Selection (Pallidum) Execution (Thalamus) Visual Buffer (Parietal) Manual Buffer (Motor) Visual Module (Occipital/etc) Manual Module (Motor/Cerebellum) External Environment CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 3 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 4

2 PM overview PM modules act in parallel can speak, type, listen simultaneously (in theory) BUT there s a central bottleneck: Cognition for perception, cognition must process information waiting at perceptual modules for motor actions, cognition must initiate movement by issuing motor command So... ACT-R can do many things at once, but can only think about one thing at a time Let s look at examples, focusing on Vision Key #1: Spotlight theory of attention what the world looks like how we see the world limited fovea of high resolution large periphery of degrading resolution CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 5 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 6 +visual-location Find a location in my visual field that satisfies a set of constraints Where Where process +visual-location Find a location in my visual field that satisfies a set of constraints =visual-location +visual =visual Given that I have the location, move my attention to that location Attend to that object What +visual-location> isa visual-location screen-x {x (greater-than x) (less-than x) (within x1 x2) lowest highest greater-than-current less-than-current} screen-y nearest {current x} attended {t nil new} :visual-num-finsts 4 items :visual-finst-span 3 (seconds) CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 7 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 8

3 Example (P find-unattended-letter ISA read-letters state start ==> +visual-location> ISA visual-location :attended nil state find-location ) (P attend-letter ISA read-letters state find-location =visual-location> ISA visual-location?visual> state free ==> +visual> ISA move-attention screen-pos =visual-location state attend ) (P encode-letter ISA read-letters state attend =visual> ISA text value =letter ==> letter =letter state respond ) Buffer stuffing ACT-R automatically puts a visual-location when visual-location buffer is empty not necessary for this homework, but you can try using it CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 9 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 10 What ACT-R sees Visual + Eye movements Key #2: Fixation visual attention Where you re looking isn t always the same as where you re attending! fixation = pause while looking (~ ms) saccade = rapid, ballistic movement (~20-30ms) eye movements follow visual attention think a rubber band! note: other types of eye movements as well (smooth pursuit, vergence) CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 11 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 12

4 Visual + Eye movements A sample task... = visual attention = fixation = cross to look at Visual + Eye movements How do our eyes move when we read? fixated cross fixation cross moves, attention on new cross My favorite animal is the platypus. eyes start to move after ~200 ms, finish ~20 ms later sometimes, corrective saccade CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 13 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 14 Visual + Eye movements How do our eyes move when we read? My favorite animal is the platypus. fixation saccade fixation etc. attention moves from word to word; fixation does not! tend to skip short or high-frequency words (e.g., a, the ) Visual + Eye movements Eye movements follow shifts of attention EMMA theory (Eye Movements & Movement of Attention) at some point after an attention shift (~200ms), the eyes move to the attended object what can happen? movement can miss target second saccade, re-fixation needed attention can shift again before movement eyes skip over first target My favorite animal is the platypus. CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 15 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 16

5 Motor module Motor movements specifically for HCI keystrokes, mouse movements, mouse clicks Cognition issue motor commands (P respond ISA letter state?manual> state ==> state +manual> ISA key ) read-letters =letter respond free done press-key =letter Motor module Timing for motor movements Preparation ~50ms per feature e.g., hand, finger, type of movement Execution: Fitts Law k = scaling constant d = distance to move w = width of target T = k log (d/w +.5) d click me! w CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 17 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 18 Other modules Audition module like Vision: cognition directs aural attention (hearing), waits for encoded chunk Speech module like Motor: cognition issues commands, module handles action independently Additional modules needed? you bet! again, currently best for HCI-centered tasks General issues How does ACT-R do things in parallel? sample execution timeline for cognition et al. Cognition Vision Attention Eye Move Motor Speech... CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 19 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 20

6 General issues But how does ACT-R really interact? Model in ACT-R Simulation Environment in LISP General issues Where do all these theories of perception, motor movement, etc. come from? typically, from other researchers! ACT-R incorporates a number of the biggest theories into a single computational framework example: visual attention spotlight idea has been around a long time ACT-R implements the spotlight so that it can be used in real models & applications we can then fit ACT-R models to existing data CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 21 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 22 Constructing Models More Realistically... What is the process? In theory: Collect a protocol / do a task analysis Write English version of production rules Create chunk types to support those rules Decide what declarative knowledge is necessary Write production rules In practice, this is not a linear process English Productions Protocol/ Task Analysis Chunk Types Declarative Memory ACT-R Productions CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 23 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 24

7 Even More Realistically... English Productions Protocol/ Task Analysis Chunk Types Domain: Subitizing Subitizing refers to people s ability to visually enumerate (i.e. count) items. Quickly count the items in the page and state the number! Declarative Memory ACT-R Productions CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 25 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 26 Domain: Subitizing Example Ready? CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 27

8

9

10 What happened? Can you explain your subjective experience of what happened Where there differences in your ability to count the items? CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 38 Was it something like this? Subitizing: Prototypical Data Peterson and Simon (2002) CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 39 CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 40