A Hierarchical Bayesian Approach to Optimal Experimental Design: A Case of Adaptive Vision Testing

Transcription

1 A Hierarchical Bayesian Approach to Optimal Experimental Design: A Case of Adaptive Vision Testing Jay Myung Computational Cognition Lab Ohio State University Joint work with Mark Pitt Woojae Kim Hairong Gu Zhong-Lin Lu Based on Psychonomic Society Meeting Presentation (Nov 18, 2016: Boston, MA)

2 Scientific Inquiry Requires Use of Tools Tools help us answer questions like How do children learn? A large chunk of science is spent on developing tools Automation is the next frontier in tool development Less involvement of humans in the drudgery of science 2

3 Cognitive Science of Old Memory drum Reel-to-reel tape deck Oscilloscope Significant human intervention in preparation, collection, and analysis of data 3

4 Cognitive Science of Today 4

5 Traditional Experimentation Theory/ Model Heuristic designs Experiment Inferences Suboptimal designs Non-adaptive (no-feedback loop) 5

6 Not All Experimental Designs Are Equally Informative : Optimal Experimental Design (OED) 6

7 Optimizing Experimental Design (OED) in Substantive Fields Statistics (Lindley, 1956; Kiefer, 1959) Vision science (Lesmes et al, 2010) Neuroscience (Lewi et al, 2009) Economics (Atkinson & Donev, 1992) Engineering (Allen et al, 2003) Systems biology (Kreutz & Timmer, 2009) Clinical drug trials (Wathen & Thall, 2008) Nano-materials (Nikolaev et al, 2014) Cognitive science (Myung & Pitt, 2009) 7

8 Autonomous Research System (ARES) for Carbon Nanotubes Synthesis at AFRL (Courtesy of the Air Force Research Lab) 8

9 Adaptive Design Optimization (ADO) (Computational Cognition Lab at Ohio State) Adaptively designed experiments Run a typical behavioral experiment as a sequence of miniexperiments/trials Optimized the design of the next mini-experiment/trial on the fly based on observed outcomes from the previous mini-experiments, so as to accelerate inference Myung & Pitt (2009) Psychological Review Cavagnaro, Myung, Pitt & Kujala (2010) Neural Computation Myung, Cavagnaro & Pitt (2013) Journal of Mathematical Psychology 9

10 Adaptive Design Optimization (ADO) Autonomous Experimentation System (closed-loop) 10

11 ADO formulated within a Bayesian decision theoretic framework t <- t+1 Posterior Bayesian Updating Optimal Design Observed Outcome Prior Design Optimization Experiment 11

12 Traditional vs. ADO Experimentation Theory/ model Heuristic designs Experiment Inferences ADO Parametric model Design optimization Experiment Inferences Optimized designs Adaptive (feedback loop) 12

13 Technical Details of ADO Next Trial/mini experiment Mutual information as the utility of design d: 13

14 Threshold Estimation of Psychometric Function 14

15 Video Demo 15

16 Current Work: Hierarchical Extension of ADO Typically, ADO starts with non-informative priors To achieve even greater efficiency, ADO can be extended to take advantage of data collected from other individuals in the same task Basic idea: Why not use the other individuals data as an informative prior for a new individual? ADO Next Trial Posterior Bayesian Updating Knowledge from other participant s data Prior Design Optimization Optimal Design Experiment Observed Outcome 16

17 Hierarchical Adaptive Design Optimization (HADO) (Kim, Pitt, Lu, Steyvers & Myung, 2014 NC) HADO combines the advantages of ADO and hierarchical Bayes modeling (HBM) to make judicious experimental designs from the very first trial. 1. ADO: Optimize based on responses from earlier trials of the current experiment 2. HBM: Utilize responses collected from other individuals from previously run experiments 17

18 HADO Framework Next Individual Posterior of (Hyper-) parameters ADO (Each Individual) Next Trial Posterior Optimal Design Bayesian Updating Observed Outcome Prior Design Optimization Experiment 18

19 HADO: Does It Work in Practice? 19

20 Empirical Validation of HADO in Adaptive Vision Testing 20

21 Testbed of HADO: Adaptive Estimation of Contrast Sensitivity Function (CSF) (Lesmes, Lu, Baek & Albright, 2010 JOV) Invisible Contrast Sensitivity Visible CSF modeled by 4 free parameters Spatial Frequency 21

22 CSF Parameterization θ = (δ, β, γ, f) Reparametrized (AULCSF, cutsf) 22

23 Hierarchical Bayes Modeling of CSF Parameters Parameter space ~ D(θ η) Contrast Sensitivity Individual 1 Individual 2 Individual n Spatial Frequency Contrast Sensitivity Spatial Frequency Contrast Sensitivity Spatial Frequency Trial 1 Trial t Trial 1 Trial t Trial 1 Trial t 23

24 HADO-based CSF Estimation Hierarchical Model Updating HBM Informative Prior Next individual s ADO Next Trial Individual Model Updating Contrast Sensitivity Session 1 Session 2 Session n Spatial Frequency Contrast Sensitivity Spatial Frequency Contrast Sensitivity Spatial Frequency Design Optimization Optimal Design Experiment Observed Outcome 24

25 HADO Demonstration and Validation (Gu et al, 2016 JOV) The benefits of HADO were demonstrated in a behavioral study in which CSFs were estimated with human participants: Phase I (Baseline Experiment): Collect data with which to build HADO-based informative priors (100 participants) Phase II (Two Validation Experiments): Demonstrate the superiority of HADO over ADO 25

26 Phase I (Baseline Experiment) ADO-based adaptive estimation of CSF with 100 participants Each participant is subject to three experimental conditions: 1 Normal: bare eyes 2 ND1: weak neutral density (wearing filtered goggles) 3 ND2: strong neutral density (wearing filtered goggles) 26

27 Phase II (Validation Experiment #1): To determine how large must be the sample size of the HADO prior to find a significant improvement over ADO? HADO informative priors of different sample sizes (Normal condition): 27

28 Take-home: The clear advantage of HADO over ADO is demonstrated in both experiment and simulation. HADO seems to work well even with priors built based on data from a small number (5-12) of participants. 28

29 Phase II (Validation Experiment #2): Effects of Prior Misspecification What happens if a wrong prior is specified? How well can HADO take advantage of group membership differences to improve parameter estimation of an individual from a designated group? How much of a cost in estimation is incurred if group membership is misspecified? 29

30 HADO priors from different experimental conditions: Diffuse Normal Nd1 Nd2 Mixture Contour of Probability Density Function Contour of Probability Density Function Sample_nor Sample_nd1 Sample_nd2 Sample_mix cutsf cutsf AULCSF AULCSF 30

31 Again, an informative prior accelerates parameter estimation. While the greatest savings are obtained with a correctly specified prior, a mixture prior is likely the best choice in practice, given its robustness against prior misspecification. 31

32 Conclusions HADO provides a judicious way to exploit two complementary schemes of inference (with past and current data) to achieve even greater accuracy and efficiency than the standard ADO. Other current & future work in lab HADO for cognitive neuroscience (e.g., fmri) HADO for delayed discounting HADO for psychotic diagnosis (e.g., OCD) HADO for nano-materials science (e.g., CNTs) 32

33 Thank You 33

34 34

35 35

36 36