How to get the most out of data with Bayes

Transcription

1 How to get the most out of data with Bayes Thomas Bayes Zoltán Dienes Harold Jeffreys

2 Evidence for H0 No evidence to speak of Evidence for H1

3 P-values make a two-way distinction: Evidence for H0 No evidence to speak of Evidence for H1

4 P-values make a two-way distinction: Evidence for H0 No evidence to speak of Evidence for H1 NO MATTER WHAT THE P-VALUE, NO DISTINCTION MADE WITHIN THIS BOX

5 Geoff Cumming: * *? * ***<? * *? ** ** * ** * H 0 µ diff Successive experiments Difference in verbal ability

6 The Bayes Factor: Strength of evidence for one theory versus another (e.g. H1 versus H0): The data are B times more likely on H1 than H0

7 From the axioms of probability: P(H1 D) = P(D H1) * P(H1) P(H0 D) P(D H0) P(H0) Posterior odds = Bayes factor * prior odds Defining strength of evidence by the amount one s belief ought to change, Bayes factor is a measure of strength of evidence

8 A model of H0

9 A model of H0 A model of the data

10 A model of H0 A model of the data A model of H1

11 How do we model the predictions of H1? How to derive predictions from a theory? Theory Predictions

12 How do we model the predictions of H1? How to derive predictions from a theory? Theory assumptions Predictions

13 How do we model the predictions of H1? How to derive predictions from a theory? Theory assumptions Predictions Want assumptions that are a) informed; and b) simple

14 How do we model the predictions of H1? How to derive predictions from a theory? Theory assumptions Model of predictions Plausibility Magnitude of effect Want assumptions that are a) informed; and b) simple

15 Example Initial study: flashing the word steep makes people walk 5 seconds more slowly done a fixed length of corridor (20 versus 25 seconds). Follow up Study: flashes the word elderly. What size effect could be expected?

16 Some points to consider: 1. Reproducibility project (osf, 2015): Published studies tend to have larger effect sizes than unbiased direct replications; Original effect size Psychology Replication effect size Behavioural economics 2. Many studies publicise effect sizes of around a Cohen s d of 0.5 (Kühberger et al 2014); but getting effect sizes above a d of 1 very difficult (Simmons et al, 2013).

17 1. Assume a measured effect size is roughly right scale of effect 2. Assume rough maximum is about twice that size 3. Assume smaller effects more likely than bigger ones => Rule of thumb: If initial raw effect is E, then assume half-normal with SD = E Plausibility Possible population mean differences

18 Example: Does imagining a sports move improve sports performance

19 Example: Does imagining a sports move improve sports performance 1. Assume real practicing move provides a plausible maximum 2. Assume population effect of imagination could be any value from 0 to that maximum 3. Assume all those possible population effects are equally plausible

20 Example Does imagining a sports move improve sports performance Plausibility Population difference in means between practice versus no practice Performance with real practice for same amount of time

21 If can determine an approximate expected size of effect Use (half) normal with SD = to that size Plausibility Expected effect Possible population mean differences -> 0 If can determine an approximate upper limit of effect => Use uniform from 0 to that limit

22 To calculate a Bayes factor must decide what range of differences are predicted by the theory 1) Uniform distribution 2) Normal 3) Half normal 4) Cauchy 5) Half-Cauchy

23 Cauchy: If scale factor r is used, maximum is about 7r Normal: if SD = r is used, maximum is about 2r

24 Example Initial study: flashing the word steep makes people walk 5 seconds more slowly done a fixed length of corridor (20 versus 25 seconds). Follow up Study: flashes the word elderly. Roughly 5 second effect expected. Is plausible maximum closer to 10 seconds or 35 seconds?

25 Example Initial study: flashing the word steep makes people walk 5 seconds more slowly done a fixed length of corridor (20 versus 25 seconds). Follow up Study: flashes the word elderly. Roughly 5 second effect expected. Is plausible maximum closer to 10 seconds or 35 seconds? So (half) normal more appropriate than Cauchy.

26 Example revised Initial study: flashing the word steep makes people walk 1 second more slowly done a fixed length of corridor (20 versus 21 seconds). Follow up Study: flashes the word elderly. Roughly 1 second effect expected. Is plausible maximum closer to 2 seconds or 7 seconds? Measure time for elderly top walk down corridor: 30 seconds Now Cauchy would be justified

27 A model of H0 A model of the data A model of H1

28 To calculate Bayes factor in a t-test situation Need same information from the data as for a t-test: Mean difference, Mdiff SE of difference, SEdiff

29 To calculate Bayes factor in a t-test situation Need same information from the data as for a t-test: Mean difference, Mdiff SE of difference, SEdiff Note: t = Mdiff / SEdiff SEdiff = Mdiff/t Also note F(1,x) = t 2 (x)

30 If B > 1 then the data supported your theory over the null If B < 1, then the data supported the null over your theory If B = about 1, experiment was not sensitive. Jeffreys, 1939: Bayes factors more than 3 or less than a 1/3 are substantial B > 3 substantial (/moderate) support for theory B < 1/3 substantial (/moderate) support for null

31 Bayes factors make the three way distinction: 0 1/3 1/3 3 3 Evidence for H0 No evidence to speak of Evidence for H1

32 To calculate a Bayes factor: 1) Google Zoltan Dienes 2) First site to come up is the right one: 3) Click on Click here for a Bayes factor calculator 4) Scroll down and click on Click here to calculate your Bayes factor!

33

34

35

36 Bayes The tai chi of the Bayes factors p The dance of the p values * *.034? *.028 ***<.001?.056 * *.024?.083 ** **.006 *.028 **.002 * H 0 µ diff Successive experiments Difference in verbal ability

37 Ways of modelling H1

38 1.Use a (similar) study to predict another (e.g. replication) 6.Take a sample to act as an original estimate 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts

39 1. Use similar effect to predict another Subliminal effect using back masking, 5%, SE = 1.5% p<.05 Subliminal effect using a new method, gaze contingent crowding : 1%,SE = 2% Is there evidence for any subliminal perception using the new method?

40 1. Use similar effect to predict another Subliminal effect using back masking, 5%, SE = 1.5% p<.05 Subliminal effect using a new method, gaze contingent crowding : 1%,SE = 2% Is there evidence for any subliminal perception using the new method? Thus: Used a half-normal with SD = 5% BF = 0.56 Nothing follows about whether or not there was subliminal perception with the new method. Need to run more subjects.

41 CANNOT USE SAME DATA TO INFORM PREDICTIONS AND ALSO TEST THE THEORY BASED ON THOSE PREDICTIONS Theory: High seating makes people feel powerful assumptions TEST Predictions: diff of 0.3 units DATA: diff of 0.3 units

42 5-min talk given to one of two groups about harm of smoking intervention no intervention Not Smoker Smoker 14 22

43 5-min talk given to one of two groups about harm of smoking intervention no intervention Not Smoker Smoker Odds ratio = (19 22) / (14 10) = 2.99

44 5-min talk given to one of two groups about harm of smoking intervention no intervention Not Smoker Smoker Odds ratio = (19 22) / (14 10) = 2.99 Ln odds ratio (1.10) is normally distributed with squared SE = 1/19 + 1/10 + 1/14 + 1/22 = 0.27 (z = 1.10/ 0.27 = 2.12, p =.034)

45 Conceptual replication: Instead of using 5-min talk, hand out pamphlet on same material intervention no intervention Not Smoker Smoker Odds ratio = (20 17) / (16 15) = 1.42 Ln odds ratio (0.35) is normally distributed with squared SE = 1/20 + 1/16 + 1/15 + 1/17 = 0.24 (z = 0.35/ 0.24 = 0.71, p =.48 )

46 Original study: ln OR = 1.10 (measure of strength of effect) Conceptual replication: Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets are not effective?

47 Original study: ln OR = 1.10 (measure of strength of effect) Conceptual replication: Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets are not effective? 1.10

48 Original study: ln OR = 1.10 (measure of strength of effect) Conceptual replication: Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets are not effective? 1.10 B H(0,1.1) = 0.78

49 Original study: Give talk and hand out pamphlet ln OR = 1.10 (measure of strength of effect) Conceptual replication: Only hand out pamphlet Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets alone are not effective?

50 Original study: Give talk and hand out pamphlet ln OR = 1.10 (measure of strength of effect) Conceptual replication: Only hand out pamphlet Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets alone are not effective? Plausibility 0 Possible population effects (Ln OR) Effect for full intervention = 1.1

51 Original study: Give talk and hand out pamphlet ln OR = 1.10 (measure of strength of effect) Conceptual replication: Only hand out pamphlet Ln OR = 0.35, SE = 0.49 non-significant Does the non-significant effect mean that pamphlets alone are not effective? B U[0,1.1] = 1.01 Plausibility 0 Possible population effects (Ln OR) Effect for full intervention = 1.1

52 General trends (in literature, other data) What are the correlates of hypnotisability? The literature gets correlations between about 0.1 or lower and up to 0.5. So for a new postulated correlate (mindfulness?) use a normal or halfnormal with an SD of 0.25 (Fisher z transform a Pearson s r to make it normal.)

53 Raw versus standardised effects: Raw: Standardised: mean difference in units of DV (seconds, scale points) Cohen s d, r Standardised effects are affected by how much noise is present in the design: i.e. by number of trials per condition, other factors in analysis Thus standardised effect sizes are often affected by theory irrelevant factors and BFs are typically best calculated on raw effect sizes. r 2 = t 2 / (t 2 + df) Fisher s z = 0.5 loge[(1 + r) / (1 - r)] (on your calculator loge may be written Ln ) SE = 1 / sqrt(df -1)

54 Making relative predictions: Find out what proportion a key effect is of a standard Peter Lush: How does performing post-hypnotically (etc) affect intentional binding? Previous study found people with conversion movement disorder had an intentional binding effect that was halved. So for all comparisons, we modelled H1 with an expected difference between groups of 50% of the intentional binding effect found in our controls

55 1.Use a similar effect to predict another 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts 6.Take a sample to act as an original estimate

56 Dienes, Baddeley & Jansari 2012: New learning paradigm Ran pilot to measure amount of learning Then in main experiment looking at effect of mood on learning, the amount of learning found in pilot was used to scale expectations of effects of mood.

57 Ziori & Dienes 2015 Gender of participant X gender of face X attractiveness of face 2 X 2 X 2 ANOVA on amount of learning Every effect scaled by overall amount of learning (6% above baseline) a B for every p

58 Watson, Gordon, Stermac, Kalogerakos, and Steckley (2003) compared Process Experiential Therapy (PET) with(cbt) in treating depression. (BDI) pre post Change CBT PET F for group (CBT vs PET) X time (pre vs post) interaction = 0.18, non-significant. 13

59 Watson, Gordon, Stermac, Kalogerakos, and Steckley (2003) compared Process Experiential Therapy (PET) with(cbt) in treating depression. (BDI) pre post Change CBT PET F for group (CBT vs PET) X time (pre vs post) interaction = 0.18, non-significant. The sample raw interaction effect is = t = 0.18 = Thus SE = 1.08/0.42 =

60 Watson, Gordon, Stermac, Kalogerakos, and Steckley (2003) compared Process Experiential Therapy (PET) with(cbt) in treating depression. (BDI) pre post Change CBT PET F for group (CBT vs PET) X time (pre vs post) interaction = 0.18, non-significant. The sample raw interaction effect is = t = 0.18 = Thus SE = 1.08/0.42 = Plausibility 0 13

61 Watson, Gordon, Stermac, Kalogerakos, and Steckley (2003) compared Process Experiential Therapy (PET) with(cbt) in treating depression. (BDI) pre post Change CBT PET F for group (CBT vs PET) X time (pre vs post) interaction = 0.18, non-significant. The sample raw interaction effect is = t = 0.18 = Thus SE = 1.08/0.42 = BU[0,13] = 0.36 Plausibility 0 13

62 Interactions > 1-df effect Congruent Incongruent Neutral Suggestion No suggestion

63 Congruent Incongruent Neutral Suggestion No suggestion Decompose interaction: suggestion X Stroop interference F(1,x) suggestion X Stroop facilitation F(1,x) Raw effect for suggestion X interference = (inc neutral) for no suggestion - (inc neutral) for suggestion = ( ) - ( ) = 25ms

64 Congruent Incongruent Neutral Suggestion No suggestion Decompose interaction: suggestion X Stroop interference F(1,x) suggestion X Stroop facilitation F(1,x) Raw effect for suggestion X interference = (inc neutral) for no suggestion - (inc neutral) for suggestion = ( ) - ( ) = 25ms SE = mean difference/t = 25 /sqrt(f) 0 50ms

65 1.Use a (similar) effect to predict another 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts 6.Take a sample to act as an original estimate

66 Zero-correlation criterion of unconscious perception: People do not know that they know when there is no relation between confidence and first-order accuracy But how strong a relationship can we expect if one did exist?

67 Knowledge is unconscious if confidence unrelated to accuracy Express accuracy as Type I d Express confidence accuracy relationship as Type II d or meta-d Given theory that Type II cannot exceed Type I Put uniform on Type II between 0 and upper limit defined by Type I Plausibility of population Type II d 0 Type I d

68 Flash: Pick dog versus Not dog ; choose between dog and cat Can people combine NOT with a word subliminally so that they are biased to pick another word? After each decision give confidence, %

69 Type I d M =.07, SE =.03, t(21) = 2.60, p =.02, People could follow the subliminal instructions

70 Type I d M =.07, SE =.03, t(21) = 2.60, p =.02, People could follow the subliminal instructions Type II d M =.01, SE =.01, t(21) = 1, p =.32. Is there no relation between confidence and accuracy?

71 Type I d M =.07, SE =.03, t(21) = 2.60, p =.02, People could follow the subliminal instructions Plausibility Type II d M =.01, SE =.01, t(21) = 1, p =.32. Type I Population Type II performance Is there a relation between confidence and accuracy? B = The data do not say. (Combining experiments together: B = 0.36.)

72 1.Replication: use one (similar) study to predict another 6.Take a sample to act as an original estimate 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts

73 How can knowledge of effect for one DV inform effect size for another? A. Does trance slow down time? Jean-Remy Martin NORMAL TRANCE 1-sec 1-sec Longer or shorter than first? Dependent variable: Percentage longer judgments How big an effect could be expected?

74 How can knowledge of effect for one DV inform effect size for another? A. Does trance slow down time? Jean-Remy Martin NORMAL TRANCE 1-sec 1-sec Longer or shorter than first? Dependent variable: Percentage longer judgments How big an effect could be expected? Past study: Trance makes people judge stimuli as 80% shorter in time

75 Calibration experiment: 1-sec 0.80 sec Proportion Longer 20% So can use this to model H1, i.e. proportions up to the full range (0 50%) were regarded as possible

76 B. Is contextual cueing based on implicit knowledge? Searching for a target; when display of distractors are repeated cueing is faster Cueing effect 200 ms, d = 2, t(12) = 6.17, p = Recognition test d = 0.16, t(11) = 0.56, p = 0.59

77 B. Is contextual cueing based on implicit knowledge? Searching for a target; when display of distractors are repeated cueing is faster Cueing effect d = 2, t(12) = 6.17, p = Recognition test d = 0.16, t(11) = 0.56, p = 0.59 Smyth & Shanks 2008 Cueing effect d = 0.5, t(78) = 4.70, p = Recognition test d= 0.6, t(39) = 3.68

78 B. Is contextual cueing based on implicit knowledge? Searching for a target; when display of distractors are repeated cueing is faster Cueing effect d = 2, t(12) = 6.17, p = Recognition test d = 0.16, t(11) = 0.56, p = 0.59 Smyth & Shanks 2008 Cueing effect d = 0.5, t(78) = 4.70, p = Recognition test d= 0.6, t(39) = 3.68 So Cohen s d should be about equal? B JZS(r = 2) = 0.13 (NB default B JZS(r =.707) = 0.329)

79 B. Is contextual cueing based on implicit knowledge? Searching for a target; when display of distractors are repeated cueing is faster Cueing effect 200 ms, d = 2, t(12) = 6.17, p = Recognition test (1 trial) recognition = 58% d = 0.16, t(11) = 0.56, p = 0.59 Smyth & Shanks 2008 Cueing effect 50 ms, d = 0.5, t(78) = 4.70, p = Recognition test (48 trials) recognition = 60%, d= 0.6, t(39) = 3.68 Plausibility 0 20 Population difference in proportion correct above 50

80 B. Is contextual cueing based on implicit knowledge? Searching for a target; when display of distractors are repeated cueing is faster Cueing effect 200 ms, d = 2, t(12) = 6.17, p = Recognition test (1 trial) recognition = 58% d = 0.16, t(11) = 0.56, p = 0.59 Smyth & Shanks 2008 Cueing effect 50 ms, d = 0.5, t(78) = 4.70, p = Recognition test (48 trials) recognition = 60%, d= 0.6, t(39) = 3.68 So using the raw recognition means B H(pc = 0.7) = 0.87 Plausibility 0 20 Population difference in proportion correct about 50

81 When the scientific problem is properly formulated, Bayesian statistics incentivize finding the correct answer

82 1.Replication: use one (similar) study to predict another 6.Take a sample to act as an original estimate 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts

83 Hypnotherapists claim some hypnotic inductions are more effective than others. Research comparing different inductions has yielded non-significant results. DV = number of suggestions passed out of 10. Traditional vs active alert: diff = 0.3 (SE = 0.3) Traditional vs idiosyncratic: diff = 1.2 (SE = 1.2) Traditional vs double induction : diff = -0.5 (SE = 0.3) Traditional vs indirect: diff =.01 (SE = 0.3)

84 Jean-Remy Martin Professional hypnotherapists surveyed and asked about the difference between the traditional induction and the other inductions 1. most likely average population difference, 2. maximum population difference that is just plausible 3. minimum population difference that is just plausible

85 Indirect Idiosyncratic Double induction Active alert

86 Indirect Idiosyncratic Double induction Active alert

87 Indirect B N( ) = 0.09 Idiosyncratic B N(1.80, 0.40) = 0.30 Double induction B N(0.62, 0.31) = 0.13 Active alert B N(1.1, 0.45) = 0.24

88 1.Replication: use one (similar) study to predict another 6.Take a sample to act as an original estimate 2.Find basic effect; use that as estimate of scale of effect for intervention 3.Theory derived limits 4.Calibration with other data 5.Survey of experts

89 Use a proportion of the data to estimate the effect for modelling H1; rest of data to test H1 against H0. What proportion? N? Need to take many such samples; average the results.

90 Use a proportion of the data to estimate the effect for modelling H1; rest of data to test H1 against H0. What proportion? N? Need to take many such samples; average the results. Violates the principle: Estimate an effect only after establishing there is an effect

91 Use a proportion of the data to estimate the effect for modelling H1; rest of data to test H1 against H0. What proportion? N? Need to take many such samples; average the results. Violates the principle: Estimate an effect only after establishing there is an effect What is the theory being tested? Why not just estimate parameter values?

92 Cumulating evidence: Study 1, study 2, study 3... What is the overall evidence for the theory? If use representation1 of the theory each time, can t just combine each individual BF: i.e. can t use BF(overall) = BF(study1) x BF(study2) X BF(study3). Why? Because each data set should also update the predictions of the theory, i.e. we should use BF(study 1 given representation1) X BF(study 2 given representation 1 + study 1).. SO Easiest way to get a BF for all data: Combine all data together first and calculate BF(all data given representation 1)

93 Study 1: mean1 SE1 Study2: mean2 SE2 Study3: mean3 SE3 Weight for study 1 =W1 = 1/SE1 2 = the precision of study 1 s estimate Overall mean = (mean1*w1 + mean2*w2 + mean3*w3)/(w1 + W2 + W3) Overall precision P = W1 + W2 + W3 Overall SE = sqrt(1/p) Can use the overall mean and SE for a) An overall BF b) An overall confidence interval (or Bayesian equivalent)

94 Study 1: mean1 SE1 Study2: mean2 SE2 Overall estimate Can use the overall mean and SE for a) An overall BF b) An overall credibility interval

95

96

97