Efficiency and design optimization

Transcription

1 Efficiency and design optimization Tuesday, Lecture 2 Jeanette Mumford University of Wisconsin - Madison Thanks to Tom Liu for letting me use some of his slides!

2 What determines your statistical power?

3 Your N is fixed what do you Give up? do? Let s look at the variance! Lowering the variance will increase our power

4 Efficient designs will increase power Lowering variance increases power

5 Efficient designs will increase power Lowering variance increases power Remember the 2 portions of variance from yesterday? The group level variance has an extra component 2 MFX = c(x 0 X) 1 c 0 2 w + 2 b

6 Efficient designs will increase power Lowering variance increases power Remember the 2 portions of variance from yesterday? The group level variance has an extra component 2 MFX = c(x 0 X) 1 c 0 2 w + 2 b

7 Efficiency E ciency = 1 c(x 0 X) 1 c 0

8 Interpreting efficiency Relative measure Rank designs and pick the one with highest efficiency (lowest variance)

9 Motivating illustration Let s assume we re working with convolved regressors More on this later! Two stimuli (positive and negative images) Stimulus duration = 1s ISI = 2s (fixed) Contrast of interest: positive negative How much can I increase efficiency by searching over random trial orders?

10 Motivating illustration Within- and between-subject variance estimates from real data Typically you won t be able to do this!! within-subject variance = 3036 between-subject variance = 810 N=30 What is the power as a function of effect size? alpha = 0.05

11 Motivating illustration Power for N=30 Power Most efficient Least efficient Effect Size

12 Motivating illustration Power for N=30 Power Most efficient 81% v 63% power Least efficient Effect Size

13 Things that might increase efficiency Random trial order Jittering your ITI Increasing your ITI Lengthen stimuli Increase number of stimuli

14 Which is the best design? It depends on the experimental question. E Slide courtesy of Tom Liu

15 What s to come What design is best? What is Estimation Efficiency vs Detection Power? You need to know to make educated decisions when choosing software What about efficiency for multiple contrasts? Is there a way to speed up the process?

16 Considerations Statistical efficiency Psychological considerations How you plan on modeling the data

17 Estimation vs Detection Pile of sand example

18 Estimation vs Detection Pile of sand example You can detect that something is there but what??

19 Estimation vs Detection Pile of sand example Zoom in and you can see a little better, but you can t be sure about what exactly is in here

20 Estimation vs Detection Pile of sand example You can sort of see something is there..but is is harder to detect

21 Estimation vs Detection Pile of sand example Zoom in and we can clearly estimate all of the shapes that are present!

22 Estimation vs Detection Detection Is there a signal present? What is the magnitude of the signal Estimation What is the shape of the fmri signal?

23 Estimation If you are interested in the shape of the HRF, is a blocked or ER design better?

24 Estimation If you are interested in the shape of the HRF, is a blocked or ER design better? Separated ER design Use the FIR model to capture shape Jitter trial onsets to fit in the most trials within a given run GLM can separate out overlap in signal

25 Detection Which design is best for detecting signal?

26 Detection Which design is best for detecting signal? Blocked

27 Estimation? Models?

28 Estimation? Models?

29 Detection? Models?

30 Models? Detection? Y=Xb+e

31 The math behind efficiency To get the highest efficiency we strive for the lowest variance for a contrast

32 The math behind efficiency To get the highest efficiency we strive for the lowest variance for a contrast c must be a vector

33 The math behind efficiency What if we have multiple contrasts? 1. Calculate c(x X) -1 c for each contrast of interest 2. Average together (c 1 (X X)-1c 1 + c 2 (X X)-1c 2 + c 3 (X X)-1c 3 )/3 3. Efficiency is inverse 3/(c 1 (X X)-1c 1 + c 2 (X X)-1c 2 + c 3 (X X)-1c 3 ) This is known as A-optimality, but there are other options

34 Which design is best? Depends on your hypothesis -Are you interested in estimation or detection?

35 Fundamental Trade-off Estimation Efficiency Periodic Semi-Random h 1 Detection Power Slide courtesy of Tom Liu

36 Maximizing efficiency What if you have multiple stimulus types? How do you choose the order? Need some type of search algorithm

37 Search algorithms Randomly try many design setups and choose one with best efficiency Would take a long time Not a very efficient efficiency search! Must also consider psychological factors

38 Permuted block design Start with stimuli blocked and randomly permute

39 Permuted block design Start with stimuli blocked and randomly permute

40 Permuted block design Start with stimuli blocked and randomly permute

41 Permuted block design Start with stimuli blocked and randomly permute

42 Permuted block design Start with stimuli blocked and randomly permute

43 Genetic Algorithm Uses the idea of genetics Two good parents should produce an even better offspring Mutations can lead to improvements

44 Genetic Algoritm

45 Genetic Algorithm

46 Improved genetic algorithm Kao et al (2008), NeuroImage Account for estimation, detection and psychological factors

47 Improved genetic algorithm

48 Options OptimizeX You ll use this in lab today! Based on convolved regressors detection? estimation?

49 OptimizeX

50 Other options Software tends to be limited in design types Eg. Say you have two types of stimuli (S a and S b ) that are always followed by response cues (R a and R b ) this can t be specified

51 Other options Write your own code! Detection or Estimation esign/start/ Detection optseq d.edu/optseq/ FIR model! (Estimation)

52 Be careful This only ranks designs All designs in your search may be suboptimal You re selecting the best among the worst You need another metric Variance Inflation Factor assesses collinearity

53 What I typically do Set up the parameters for the design Write my code for generating designs Generate a few designs and check the VIF If VIF is good, I run a full efficiency search

54 Questions?