Inference beyond significance testing: a gentle primer of model based inference

Transcription

1 Inference beyond significance testing: a gentle primer of model based inference Junpeng Lao, PhD Fribourg Day of Cognition 2017/10/04

2 An old tale: the problem of p-value

3 new battle

4

5 The great divide in statistical inference is not between Frequentist and Bayesian, but between those doing test versus those building models.

6 Inference beyond significance testing Take home message: Build (Bayesian) probabilistic model Think generatively Care about the estimation (validate the fit) Prediction, Prediction, Prediction Make realistic assumption (more is more) A small primer of how to do statistical inference by building probabilistic model.

7 Ugly inference using tests

8 Box s loop First gather data from some real-world phenomena. Then cycle through Box s loop (Blei, 2014). 1. Build a probabilistic model of the phenomena. 2. Reason about the phenomena given model and data. 3. Criticize the model, revise and repeat. Beautiful inference using models. Blei, D. M. (2014). Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, 1, Box, G. E. (1976). Science and statistics. Journal of the American Statistical Association, 71(356),

9 A model-based Inference workflow Blei, D. M. (2014). Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, 1, Box, G. E. (1976). Science and statistics. Journal of the American Statistical Association, 71(356),

10 Example 1: BEST In [1]: scipy.stats.ttest_ind(y1, y2, equal_var = False) Out[1]: Ttest_indResult(statistic=5.466, pvalue=5.129e-07) Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573.

11 Data generation process In [2]: scipy.stats.norm.rvs(loc=101, scale=1., size=100)

12 Data generation y ~ Gaussian(mean, std) mean is between - and + std is a positive real number Model y ~ Gaussian(mean, std) mean ~ Uniform(95, 130) std around 5.2 Bayesian Model

13 Something to generate the observed data: with pm.model() as model: # drug μ1 = pm.uniform('μ1', lower=95., upper=130.) σ1 = pm.uniform('σ1', lower=0., upper=10.) group1 = pm.normal('drug', mu=μ1, sd=σ1, observed=y1) # placebo μ2 = pm.uniform('μ2', lower=95., upper=130.) σ2 = pm.uniform('σ2', lower=0., upper=10.) group2 = pm.normal('placebo', mu=μ2, sd=σ2, observed=y2)

14 Model with pm.model() as model_flat: # drug μ1 = pm.flat('μ1') σ1 = pm.halfflat('σ1') group1 = pm.normal('drug', mu=μ1, sd=σ1, observed=y1) # placebo μ2 = pm.flat('μ2') σ2 = pm.halfflat('σ2') group2 = pm.normal('placebo', mu=μ2, sd=σ2, observed=y2)

15 You can get more from a Bayesian model

16 BEST

17 Is the difference meaningful? e.g., Effect size? with pm.model() as model: diff_of_means = pm.deterministic('μ1-μ2', μ1 - μ2) diff_of_stds = pm.deterministic('σ1-σ2', σ1 - σ2) effect_size = pm.deterministic('effect size', diff_of_means / np.sqrt((σ1**2 + σ2**2) / 2))

18 It s better be uncertain of a right answer than certain about a wrong answer. I prefer uncertainty

19 Lee and Wagenmakers' Bayesian Cognitive Modeling - A Pratical Course Chapter 8.3 Reparameterization

20 Extension: Linear Model The big linear model family a wider class of linear model repeated measurement ANOVA and HLM are specific cases of LMM

21 Generalised Linear Model y = Xβ + ε, E(y) =X = µ, E(Á) =0, cov(y) =cov(á) = 2 V.

22 Example 2: Hierarchical Linear Regression Gelman et al.'s (2007) radon dataset Gelman et al. s (2007) radon dataset is a classic for hierarchical modeling. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil. Here we ll investigate this differences and try to make predictions of radonlevels in different counties based on the county itself and the presence of a basement. In this example we ll look at Minnesota, a state that contains 85 counties in which different measurements are taken, ranging from 2 to 116 measurements per county.

23 Conventional approaches: Complete pooling or No pooling The EPA did a study of radon levels in 80,000 houses. Two important predictors: measurement in basement or first floor (radon higher in basements) county uranium level (positive correlation with radon levels)

24 Conventional approaches: Complete pooling or No pooling Response Response Predictor Predictor

25 Neither of these models are satisfactory: - if we are trying to identify high-radon counties, pooling is useless - we do not trust extreme unpooled estimates produced by models using few observations

26 Multilevel (hierarchical) models Partial pooling

27 Multilevel (hierarchical) models Partial pooling

28 Multilevel (hierarchical) models Partial pooling

29 Example 3: Smoothing

30 What is smoothing, really?

31

32 Making prediction

33 Making prediction

34 Making prediction

35 A few things to keep in mind: Modelling is natural Cross-validation Reconstruct the quantity of interested, inspect their uncertainty Model fitting is not automatic Writing down the model doesn't means you can fit it Don t throw away information! (Prior matters)

36 Tool for building modern Bayesian Model mc-stan.org docs.pymc.io

37 Further reading

38 All models are wrong, but some are useful. George E. P. Box An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem. John Tukey Reference: Box & Draper (1987), Empirical model-building and response surfaces, Wiley, p. 424.