The role of surprise in theory testing A case study from music cognition

Transcription

1 The role of surprise in theory testing A case study from music cognition Henkjan Honing ILLC CSCA University of Amsterdam

2 Thanks to Institute for Logic, Language & Computation (ILLC) Cognitive Science Center Amsterdam (CSCA) Royal Netherlands Academy of Arts and Sciences (KNAW) 2

3 Outline Computational modeling of music cognition Case study How to select among alternative models? What makes a model surprising? Discussion 3

4 Computational modeling Music Knowledge exhibits Behavior Stimuli models agrees Machine Algorithm exhibits Behavior 4 Desain & Honing (1995/2004)

5 Computational modeling Mind Mental Process exhibits Behavior Stimuli models agrees Machine Algorithm exhibits Behavior 5 Desain & Honing (1995/2004)

6 Computational modeling Mind Mental Process exhibits Behavior Stimuli models validates agrees Machine Algorithm exhibits exhibits Behavior 6 Desain & Honing (1995/2004)

7 Model selection methods A. Measure of goodness-of-fit (e.g. Rodgers & Rowe, 2002) B. Measure of simplicity (e.g. Pitt, Myung, & Zhang, 2002) C. Measure of surprise (cf. Popper, 1963) D. Ground-truth verification (e.g. Rijsbergen, 1979) E.... 7

8 Case study Computational models of expressive timing in music performance (e.g., Sundberg & Verillo, 1980; Kronman & Sundberg, 1987; Longuet-Higgins & Lisle, 1989; Feldman, Epstein & Richards, 1992; Todd, 1992; Epstein, 1994; Todd, 1995; Friberg & Sundberg, 1999; Large & Palmer, 2002) Modeling the Final Ritard: Typical slowing down at the end of a music performance (e.g., Hudson, 1996; Clarke, 1999; Gabrielsson, 1999) 8 Honing (2006)

9 Two computational approaches Kinematic approach (K model) Predicts shape of expressive timing patterns and how they conform to the laws of physical motion (commonality) Perception-based approach (P model) Predicts the amount of expressive freedom a performer has in the interpretation of a rhythmic fragment before being misinterpreted as an altogether different rhythm (diversity) 9

10 10 Repp (1992)

11 K model v(t)=u+at (1) v(x)=(u 2 +2ax) 1/2 (2) v(x)=[1+(w q -1)x] 1/q (3) 11

12 Mechanical version of K model 12

13 Mechanical version of K model 13

14 Mechanical version of K model 14

15 P model Two components: - Model of perceived regularity (tempo tracker) (Large & Jones, 1999; Toiviainen, 1999) - Model of rhythmic categorization (quantizer) (Longuet-Higgins, 1987; Desain & Honing, 1989) 15

16 Effect of rhythm and tempo on predictions 16 Honing (2005)

17 A. Measure of good fit 17 Honing (2006)

18 Conclusion of method A Measures of GOF only assess fit GOF is not able to distinguish between variations in the data caused by noise and those that the model was designed to capture Even if one model would have a significant better fit we could not select that model over the other 18

19 B. Measure of flexibility Response area : Range of possible predictions K model P model 19 Honing (2005)

20 B. Measure of flexibility Effect of note density and rhythmic structure P model 20 Honing (2005)

21 Conclusion of method B Both models making roughly similar fits to the data K model simpler than the P model However, the P model show less flexibility, and should hence be preferred 21

22 Conclusion of method B Both models making roughly similar fits to the data K model simpler than the P model However, the P model show less flexibility, and should hence be preferred Still, we can wonder how surprising all this is in the context of the phenomenon modeled 22

23 C. Element of surprise 23 Honing (2006)

24 C. Element of surprise 24 Honing (2006)

25 Towards a measure of surprise Confirmations [of a theory] should count only if they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory an event which would have refuted the theory. (Popper, 1963:47) 25

26 Towards a measure of surprise Correct prediction of an unlikely event is more surprising than the correct prediction of something that was expected anyway Prefer the model that: Minimizes the intersection of Hpredicted with respect to Hplausable While preferring the Hpredicted that is least smooth Honing & Romeijn (in prep.)

27 Towards a measure of surprise Prefer the model that: 1. Fits the empirical data well (best fit) 2. Makes limited range predictions (least flexible) 3. Makes unexpected predictions (most surprising) 27

28 Epilog Meeting a friend in the corridor, Wittgenstein said: Tell me, why do people always say it was natural for men to assume that the sun went round the earth rather than that the earth was rotating? His friend said: Well, obviously, because it just looks as if the sun is going round the earth. To which the philosopher replied, Well, what would it have looked like if it had looked as if the earth was rotating? 28 Tom Stoppard, Jumpers, 1972

29 Epilog Meeting a friend in the corridor, Wittgenstein said: Tell me, why do people always say it was natural for men to assume that the sun went round the earth rather than that the earth was rotating? His friend said: Well, obviously, because it just looks as if the sun is going round the earth. To which the philosopher replied, Well, what would it have looked like if it had looked as if the earth was rotating? 29 Tom Stoppard, Jumpers, 1972

30 References Desain, P., & Honing, H. (2004). Final Report NWO-PIONIER Project "Music, Mind, Machine". Technical Notes ILLC, X Honing, H. (2005). Music Cognition: Theory Testing and Model Selection. Proceedings of the XXVII Annual Conference of the Cognitive Science Society (CogSci2005), 38, Stresa: University of Turin Honing, H. (2006) Computational modeling of music cognition: A case study in model selection. Music Perception, 23(5), Honing, H. (2007) Preferring the best fitting, least flexible, and most surprising prediction: Towards a Bayesian approach to model selection in music cognition. Proceedings of the Society for Music Perception and Cognition (SMPC), 37, Montreal: Concordia University Roberts, S. & Pashler, H. (2000) How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), Pitt, M.A., Myung, I.J., & Zsang, S. (2002) Toward a method of selecting among computational models of cognition. Psychological Review, 109(3), Pitt, M.A., & Myung, I.J. (2002) When a good fit can be bad. Trends in Cognitive Science, 6, Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge. London: Routledge Rodgers, J.L., & Rowe, D.C. (2002) Theory development should begin (but not end) with good empirical fits: A comment on Roberts and Pashler (2000). Psychological Review, 109(3), Van Rijsbergen, C. (1979) Information Retrieval. London: Butterworth