Qualitative Model Comparisons. Q550: Models in Cognitive Science Lecture 4

Transcription

1 Qualitative Model Comparisons Q550: Models in Cognitive Science Lecture 4

2 Comparing Models We further science by constraining between different models formalize and reject models based on data Ideally, we would love to do a qualitative comparison between models, that is, a comparison of formalized principles: Ø free from ad hoc assumptions Ø independent of specific model parameters

3 Our ideal situation: Input Theory 1 A > B Input Theory 2 A < B Input Human A > B Theory 1 is consistent w/ empirical data, whereas Theory 2 is not. Further, Theory 2 cannot produce the result for any set of parameters, whereas, Theory 1 produces the result for all parameter values and assumptions of course, this rarely occurs in practice

4 Comparing Models of course, this rarely occurs in practice Often, we look at the proportion of total parameters that would produce the observed effect OR: we find the best fitting set of parameters (estimated from the data) and compare the models quantitatively

5 Comparing Models Levels of model comparison: Ø Pure qualitative comparison independent of specific parameter values Ø Proportion of total parameter that produce the effect (signed-difference test) Ø Quantitative comparison to determine which model is most likely to have produced the data (assuming all models predict the ordinal differences) Ø Generalizing to other paradigms Minimize task-specific parameters -- when these are changed, model should generalize to other tasks w/ the same general domain

6 A Qualitative Comparison in Minerva We have an existence proof: It is unnecessary to posit both an abstract and episodic memory But: do we have any reason to prefer one over the other? Could we produce the data as well using a single prototype per category as well as multiple episodes? Ø Let s keep everything the same in Minerva, but instead of storing multiple episodes, we ll create a composite for each category as they are learned, and will just store these three prototypes

7 What happens? A prototype version of Minerva cannot explain the immediate benefit for old exemplars It also cannot explain the interaction (crossover) between old exemplars and the prototype as a function of forgetting Ø A prototype version of Minerva would not explain the ordinal differences in conditions b/c we would lose differential activation of stored episodes

8 List Learning: Let s consider a second example: we re interested in learning items in a sequence and recalling them (in the order presented, or in any order) Random series of words: 1 word per second. Remember the words and recall them after the presentation You may be asked to recall them in the order presented, or in any order you choose

9 magazine keyboard fountain camera ******** gravel giraffe paper angel door +

10 Free Recall: What are we doing? Rehearsing? We ll call this a strength account Each goes in once? Decay account? Associative chaining? (S-R theory) Combinations? Serial position curves: How would these accounts produce them? If a model does not predict R > P > M then reject it If our models all make the same ordinal predictions, then we move onto qualitative model comparisons

11 Theories of Forgetting Forgetting: inability to retrieve information from memory Complicated by three processes: ENCODING -- STORAGE -- RETRIEVAL Any of these could be responsible for failure to get something out of memory (or a combination) Likely Truth: different types of info are lost by different sources How do we remember a list of items? M t = M t 1 + t i=1 I i M t = αm t 1 + I i

12 Decay Theory: The strength of a memory trace decays over time as iron rusts with time (McGeoch, 1932) Power law: Strength of trace decays as a power function of the retention interval (both at the neural and behavioral levels) Thorndike s Law of Disuse Old Decay: M t = αm t 1 + I i But, decay should be dependent on time New Decay: At time t; for i = 1 to N : M i = e λt Δ M i Where t Δ = t t encode

13 Exponential: f (x) = e λx x 0 λ > 0

14 Rust and memory decay: time is not the causal agent for either Forgetting may be a power function of time, but decay is confounded with interference: As time goes on, more things are stored, increasing the probability of retrieval failure

15 Interference Theory: Forgetting is a result of retrieval failure Jenkins & Dollenbach (1924): After identical time delays, Ss who slept after learning recalled more than those who remained awake (fewer intervening activities = less interference) PI: prior learning interferes with recall of newer information RI: new learning interferes with recall of prior information This is difficult to put in an equation, but we can still formalize it

16 Consolidation Theory: Information is not forgotten, but may not be encoded if it is not consolidated. After presentation, a period of inactivity is required (perseveration). The longer this period, the more likely the trace will be consolidated. If a trace is consolidated, it does not decay, and retrieval is perfect Ebbinghaus, retrograde amnesia, ECS, sleep Predictions: 1) A period of inactivity is more conducive to consolidation than activity 2) If perseveration is interrupted, the trace cannot be consolidated and will not be stored 3) If consolidation is prevented, the item should not be retrieved b/c storage was not completed

17 M i = M i 1 + I i M i = M i 1 + (1 e λt )I i M i = M i 1 + bin(uniform, 1- e λt )I i Where bin returns 1 if uniform 0 if uniform < 1- e λt 1- e λt If (persev) then: M i = M i 1 + bin(uniform, 1- e λt )I i Else: M i = M i 1 + bin(uniform, e λt )I i

18 Forgetting may be due to: Encoding failure (Consolidation) Storage failure (Decay) Retrieval failure (Interference) If we look at only STM, both consolidation and decay predict rapid decay with time if S is cannot rehearse or consolidate Decay is linked to time, not necessarily number of items Interference doesn t care about time, just number of items

19 Waugh-Norman paradigm: Present digits/words; S learns successive associations. At end of list, S is given an item that was in the list and recalls the item that followed it. We ll vary presentation speed 1 item/sec vs. 2 items/sec Assume items have some perceptual similarity (XGH, KCF) Assume perfect association: memory is a matrix where each vector is S//R Vector elements are binary Selection rule is the same for both, only memory changes p(r S) = D i=1 M i

20 Decay: At item i, for j = 1 to i: Where: t Δ = t i t encode M j = e λt Δ M j Interference: A new item can overwrite any shared features with an existing item with a random probability t Item presented Memory

21 1 item/second COGNITIVE MODELING HOTDOG YELLOW CARPET SWORD GUITAR MISSLE MOUSE GLOVE IGLOO VIPER ##### TAPE SHIP AXE ELF SHIP

22 4 items/second SUITCASE SCREEN ISLAND RAZOR CRAFT PLUTO ###### PIANO ROBIN TIGER TABLE BOOK GEAR DEEP LAMP SINK ICE TABLE

23 An argument: Instances vs. sets and averages: Ancient theory of eidola: things fly into your head Plato/Aristotle: Essential elements [+wings] [+beak] [+flies] Instances vs. Protypes: Rosch/Lakoff: Prototypes, basic levels have special processing advantage Brooksian: Inflexible to only store prototype Exemplar-based vs. Prototype models: Exemplars: Hintzman, Nosofsky, Prototypes: Reed

24 Typicality Effects " Rosch (1973, 1975): Typical members of a semantic category can be processed more efficiently than atypical ones RT (ms) Rosch Prototype Label Euclidean Distance High Med Low Typicality

25 (Lets use novel stimuli so we have control) Galli Tasio Galli Internal Representation Tasio Radok Samar Radok Internal Representation Samar Prototype model vs. exemplar model

26 Formalizing a model: 1) Conceptual Theory --> Formal Model 2) Ad-hoc assumptions: Representation: Random vectors for Greebles Process: Prototype averages within each category Recency? Exemplar: Instances are each stored with pattern label Similarity is Euclidean distance in both: There is randomness in the response process d ij = (x im x jm ) m M

27 Formalizing a model: 1) Conceptual Theory --> Formal Model 2) Ad-hoc assumptions 3) Parameters: Fixed: Task: num_per_cat, trials, etc. Free: Cognitive: num_features, n_flip, etc. Recency weighting, response sensitivity, distance scaling, attention When we re done, the models make precise predictions, and the only difference between models should be the theoretical debate

28 Formalizing a model: In both models: Psychological distance (Nosofsky, 1986): Similarity scaling (Shepard, 1987): d ij = (x im x jm ) m M s ij = e λd ij Decision is based on Luce (1959) choice rule: Prototype Model: P(A x) = s xpa s xpa + s xpb Exemplar Model: P(A x) = s xa a A s xa + s xb a A b B

29 Formalizing a model: They both behave like humans with the right parameter values: GCM_vs_Prototype.m OK, let s scrap linear categories, and try XOR category structure: XOR_Structure.m

30 They all work what now? Our models are based on simple units corresponding to a small number of elementary principles of cognition Once we put several of these units together, we get a complex system, and we usually require computer simulations to derive predictions from the system It becomes increasingly difficult to create qualitative tests of models as they increase in complexity Quantitative model comparisons require comparing the quantitative accuracy of the models based on the optimal set of parameters

31 They all work what now? Quant comparisons are challenging b/c models differ in complexity Ø Complexity is based on type and number of assumptions, and number of model parameters Ø Selection of the best model must satisfy a balance of both accuracy and parsimony Ø Saturated model = perfect accuracy; null model = perfectly parsimonious; but both are wrong This is not yet a fine-grained art the literature is full of debate on how this should be accomplished

32 Accuracy Parsimony Saturated model? Overfitting?

33 Occam s Razor entia non sunt multiplicanda praeter necessitatem entities should not be multiplied beyond necessity Of two equivalent theories, all other things being equal, the simpler one is to be preferred Is this helpful? Significance of variance predicted w/ added parameters More parameters must explain more variance in the data

34 Additional Criteria: 1. Accuracy of prediction 2. Parsimony of model 3. Explanatory power of parameters/assumptions 4. Generalizability to other tasks Difficult to consider all

35 Fit is not everything These techniques allow us to compare non-nested models that differ in complexity, but as Roberts and Pashler (2000) note, this may not be a good test of a model Beware of overfitting N-fold cross-validation: estimate parms from part of the data, and make predictions for the other part (bootstrap) Generalization methods: estimate parms from one experimental condition, and make predictions for the other

36 Roberts & Pashler, 2000