Introduction to Cognitive Modeling: Architectures and Approaches. Q550: Models in Cognitive Science Lecture 2

Transcription

1 Introduction to Cognitive Modeling: Architectures and Approaches Q550: Models in Cognitive Science Lecture 2

2 Review Ø We have a foggy idea what we re supposed to be doing: no text, no universal definition of a model, no accepted practice for designing and evaluating models Ø Ø Ø BUT: Over 85% of new pubs involve cognitive modeling Features of a Cognitive Model: 1. Goal is to explain basic cognitive processes 2. Described in formal languages 3. Derived from basic principles of cognition Advantages of Cognitive Models: 1. Produce logically valid predictions 2. Capable of making precise quantitative predictions 3. Capable of generalizing

3 Computational modeling requires the researcher to be explicit about a theory in a way that a verbal theory does not It removes the free wiggle room that comes with all verbal theories - Murphy (2011) Over the Christmas Holiday, Alan Newell and I invented a thinking machine - Herb Simon, January 1956

4 First Caveat: All models are wrong Regardless of their form or function, or the area in which they are used, it is safe to say that these models all have one thing in common: They are all wrong -MacCallum (2003) Modeling has often been misunderstood and incorrectly characterized, because scientists have a natural tendency to act as if models are either right or wrong, engaging in a kind of scientific combat to establish truth. Of course, none of our models are ever correct, even when restricted to the simplest and most controlled experimental settings. -Shiffrin (2010) Maps as models

5 A Model of a system:

6 A Model of a system: Components: Formal description Structure and function Direction (cause/effect) Assumptions Parameters Goals: 1. Description/Prediction 2. Process discrimination 3. Explanation

7 A Model of a system:

8 A Model of a system:

9 Another (weird) pattern of data: Nosofsky, R. M. (1991). Tests of an exemplar model for relating perceptual classification and recognition memory. Journal of Experimental Psychology: HPP, 17, 3-27.

10 Another (weird) pattern of data: Nosofsky, R. M. (1991). Tests of an exemplar model for relating perceptual classification and recognition memory. Journal of Experimental Psychology: HPP, 17, 3-27.

11 What is Cognitive Modeling? How do we differentiate a cognitive model from a conceptual or statistical model? Cognitive science is concerned with understanding the processes that the brain uses to accomplish complex tasks including perceiving, learning, remembering, thinking, predicting, inference, problem solving, decision making, planning, and moving around the environment. The goal of a cognitive model is to scientifically explain one or more of these basic cognitive processes, or explain how these processes interact. --Busemeyer & Diederich (2010)

12 Cognitive Processes Cognition: The process of thought? cf. a physical process (e.g., gravity), we want to model a cognitive process (e.g., forgetting) Traditionally, we are interested in the how, not where However, the where constrains the how: Structure and function: Inseparable (?); structure constrains what can be computed

13 Modeling an intelligent system: What is going on in Homer s head? (Digit1 - Digit2 + Digit3 - Digit4 ) - 1 If the input is positive, Homer adds the odd digits (read left to right), and subtracts for i = even 1, len(string) digits, and do then subtracts 1 at the end to be safe. So, given the input abcd, Homer would output a-b+c-d-1. If the input is negative, Homer subtracts the odd digits, end and adds the even digits, subtracting 1 at the end (to be safe). So, given the input abcd, Homer would resp output = -a+b-c+d-1. resp - 1 resp = resp + string(i) * (2*(mod(i,2)-1) If ±...edcba is an integer such that a is the digit in the ones place, b the digit in the tens place, c the digit in the hundreds place, etc., then ±...edcba returns -1 ± (a - b + c - d + e -...) if there are an odd number of digits, and returns -1 ± (- (a - b + c - d + e -...)) if there are an even number of digits. y = N i=1 x i ( 1) i 1 1

14 Features of a Cognitive Model: 1) Goal is to scientifically explain basic cognitive processes 2) Described in formal (mathematical or computer) languages VCTs are broadly stated theoretical assumptions Cognitive models differ in that they are formally stated E.g., Craik and Lockhart s LOP hypothesis provides a conceptual framework for memory, whereas SAM or TODAM, being formal, are examples of cognitive models We can convert a conceptual framework into a cognitive model by formalizing it (recasting the verbal statements into mathematical or computer language)

15 Features of a Cognitive Model: 1) Goal is to scientifically explain basic cognitive processes 2) Described in formal (mathematical or computer) languages 3) Derived from basic principles of cognition Statistical models are applicable to data from any field, as long as those data meet the statistical assumptions (normality, etc.) These assumptions are not derived from any principles of cognition, and may be inconsistent with known facts of cognition Ø E.g., statistical assumptions are inconsistent w/ RT dists Ø Cognitive models of response time accommodate this Statistical vs. explanatory models (CSLR/SVM car) Parameters should have a meaningful cognitive interpretation

16 Advantages of Cognitive Models: 1) Guaranteed to produce logically valid predictions This is not true of conclusions based on intuitively-based verbal reasoning E.g., early categorization research argued against exemplar models b/c transfer to a prototype was better than any of the experienced exemplars But once the exemplar model was formalized, it naturally produces this effect Reasoning from a conceptual framework can lead to incorrect conclusions (Goat game)

17 Advantages of Cognitive Models 1) Guaranteed to produce logically valid predictions 2) Capable of making precise quantitative predictions Even if a model predicts a correct qualitative (ordinal) effect, its predictions may be an order of magnitude off the mark Formal models allow us to evaluate models both qualitatively and quantitatively in their correspondence to empirical data Gallileo s balls

18 Advantages of Cognitive Models 1) Guaranteed to produce logically valid predictions 2) Capable of making precise quantitative predictions 3) Capable of generalizing Both cognitive models and statistical models (empirical curve fitting) are capable of generating quantitative predictions But cognitive models should also generalize to new paradigms, and should describe the process that produces the output, rather than just describing the output itself Newell s (1981) power law vs. Logan s (1988) instance theory Ø Logan s theory produces and explains the power law, and can be used to make new predictions: how variance of RT dist changes w/ practice and how accuracy changes w/ practice

19 Classifying Models: Lewandowsky & Farrell (2011) 1. Data Description (input-output function) 2. Process Identification/Characterization (peek inside black box; what processes are producing inputoutput) 3. Process Explanation (explain how black box is producing output from input)

20 1. Descriptive Models Ø Statistical models are primarily descriptive/predictive Ø Primary goal is a reduced description of the data at hand Ø Practice Functions: When learning a new skill, are time savings from practice following a power law of practice? Note: Choice of correct descriptive model can have important implications about the psychological nature of a phenomenon (here, learning) Heathcote, Brown, & Mewhort (2000): The power law repealed: The case for an exponen<al law of prac<ce. Psychonomic Bulle/n & Review, 7,

21 RT = t β RT = e αt

22 RT = t α One attribute of descriptive models is that they are RT = e βt explicitly devoid of psychological content; for example, although the existence of an exponential practice function constrains possible learning mechanisms, the function itself has no psychological content. It is merely concerned with describing the data. - Lewandowsky & Farrell (2011)

23 2. Characterization Models Ø Ability to identify cognitive constructs producing a phenomenon Ø They postulate and measure processes, but don t offer a detailed explanation of how the process works Ø Unlike descriptive models, explanatory power rests on hypothetical constructs within the mind rather than within the data to be explained (L & F, 2011). Ø However, these models do not go beyond identification of these processes they are agnostic about explaining how the processes work (e.g., LOP) Ø Example: Multinomial Processing Trees (MPTs): E.g., Batchelder & Riefer, 1999

24 3. Explanatory Models Ø Most detailed view inside the black box Ø Also posit hypothetical cognitive constructs, but with a detailed explanation of how these processes might work Ø Why bother with characterization; why not just explain everything? Ø Not always possible; not always useful Examples with Nosofsky s (1986) GCM and Hintzman s (1986) MINERVA 2

25 Steps of Cognitive Modeling: 1) Conceptual theory formal description

26 Steps of Cognitive Modeling: 1) Conceptual theory formal description 2) Ad hoc assumptions to complete formal description Conceptual theory is often missing important details, so we have to make additional detailed assumptions in order to complete the model and to generate quantitative predictions E.g., assumptions about features to represent stimuli We try to minimize the number of ad hoc assumptions, but this step is often unavoidable

27 Steps of Cognitive Modeling: 1) Conceptual theory formal description 2) Ad hoc assumptions to complete formal description 3) Parameter estimation Models almost always contain coefficients that are initially unknown, and these values need to be estimated from some of the observed data E.g., the importance weight assigned to each feature is a free parameter that is estimated from the choice response data We try to minimize the number of model parameters, but this is usually a necessary and important step

28 Steps of Cognitive Modeling: 1) Conceptual theory formal description 2) Ad hoc assumptions to complete formal description 3) Parameter estimation 4) Compare predictions to empirical data Qualitative: Ordinal and parameter free (models make these predictions for any value of the free parameters) Quantitative: Magnitude of correspondence to data Cognitive models can be compared to each other quantitatively, or we can use a base and saturated model

29 Steps of Cognitive Modeling: 1) Conceptual theory formal description 2) Ad hoc assumptions to complete formal description 3) Parameter estimation 4) Compare predictions to empirical data 5) Iterate to constrain Experiments are designed based on models, and models are constrained and modified/extended based on new data This should produce an evolution of models that improve and become more powerful over time as the science in a field progresses

30 Evolution of Math Modeling Estes, W. K. (1975). Some targets for mathematical psychology. Journal of Mathematical Psychology, 12, In looking back on math modeling, Estes notes: We lack cumulative progress (nobody builds on models, they just build new ones) We re modeling tasks, not general processes Math psyc isn t helpful to any applied problems We need to start archiving data We can t just fit a descriptive model, we need an explanatory and generalizable model

31 Evolution of Math Modeling We need agreed upon methods for model comparison and fit We fit models to highly controlled experiments, and we may be unknowingly hold constant factors which would have a major effect if allowed to vary (Ecological validity vs. control --> Largescale models) These computer things can t be trusted

32 Why computers are dangerous Computers allow us to create theory almost completely bypassing both the opportunities for ingenuity and the quantities of blood, sweat, and tears that used to go into this enterprise. (p. 268) This came true, and perhaps a bigger problem now: You don t need wisdom to whip up a model. (Youth creating theory) Is mathematical psychology in danger of becoming obsolescent before reaching maturity? (p. 279) If you were to write a Targets for cognitive models paper today, what wishes would you come up with? Lewandowsky, S., & Heit, E. (2006). Some targets for memory models. Journal of Memory and Language, 55,

33 An argument: Instances vs. sets and averages: Ancient theory of eidola: things fly into your head Plato/Aristotle: Essential elements [+wings] [+beak] [+flies] Instances vs. Protypes: Rosch/Lakoff: Prototypes, basic levels have special processing advantage, cognitive economy Brooksian: Inflexible to only store prototype Exemplar-based vs. Prototype models: Exemplars: Hintzman, Nosofsky, Prototypes: Reed

34 Typicality Effects " Rosch (1973, 1975): Typical members of a semantic category can be processed more efficiently than atypical ones RT (ms) Rosch Prototype Label Euclidean Distance High Med Low Typicality

35 (Lets use novel stimuli so we have control) Galli Tasio Galli Internal Representation Tasio Radok Samar Radok Internal Representation Samar Prototype model vs. exemplar model

36 Formalizing a model: 1) Conceptual Theory --> Formal Model 2) Ad-hoc assumptions: Representation: Random vectors for Greebles Process: Prototype averages within each category Recency? Exemplar: Instances are each stored with pattern label Similarity is Euclidean distance in both: There is randomness in the response process d ij = (x im x jm ) m M

37 Formalizing a model: 1) Conceptual Theory --> Formal Model 2) Ad-hoc assumptions 3) Parameters: Fixed: Task: num_per_cat, trials, etc. Free: Cognitive: num_features, n_flip, etc. Recency weighting, response sensitivity, distance scaling, attention When we re done, the models make precise predictions, and the only difference between models should be the theoretical debate

38 More Background Sun, R. (2008). Introduction to computational modeling. Cambridge Handbook of Computational Psychology. Cambridge University Press. Estes, W. K. (2002). Traps in the route to a model of memory and decision. Psychonomic Bulletin & Review, 9, The OSU Cognitive Modeling Repository

39 Next Class An existence proof: Hintzman, D. L. (1986). 'Schema abstraction' in a multipletrace memory model. Psychological Review, 93, Read some history and a modern manifesto: McClelland, J. L. (2009). The place of modeling in cognitive science. Topics in Cognitive Science, 1, Addyman, C., & French, R. M. (2012). Computational modeling in cognitive science: A manifesto for change. Topics in Cognitive Science, 4,