Is Random Sampling Necessary? Dan Hedlin Department of Statistics, Stockholm University

Transcription

1 Is Random Sampling Necessary? Dan Hedlin Department of Statistics, Stockholm University

2 Focus on official statistics Trust is paramount (Holt 2008) Very wide group of users Official statistics is official Bias important Not as cost sensitive as market research Strong emphasis on generality and precision (tradeoff in model building between generality, realism and precision, see Levins 1966 and Baker et al 2013 sec 8.2) 10/3/2015 NTTS Dan Hedlin, Stockholm University 2

3 Trends Wider, Deeper, Quicker, Better, Cheaper (Holt 2007) Increasing rates of nonresponse, hard to find, hard to contact in the Western world Expansion of data sources, data collection methods; mixed modes Expanding research, mostly application driven. 10/3/2015 NTTS Dan Hedlin, Stockholm University 3

4 On bias In official statistics, a minimum MSE estimator is not necessarily desirable. Bias is not on the same footing as variance Note that point estimates, not interval estimates, are used Bias is (potentially) worse than variance Also an issue of trust 10/3/2015 NTTS Dan Hedlin, Stockholm University 4

5 On balanced samples What is balance? x Basically that this holds: j = x j s n U for some N number j (ignoring weights). (Valliant et al. 2000) Sample balance Or x s = x r Response set balance 10/3/2015 NTTS Dan Hedlin, Stockholm University 5

6 A balanced sample is good to have y s y r = x s x r β r + β s β r x s (Särndal & Lundquist 2014) Does small x s x r imply small β s β r? The answer is probably yes (Särndal & Lundquist 2014, Sec. 6) 10/3/2015 NTTS Dan Hedlin, Stockholm University 6

7 So what is the causal mechanism small x s x r -> small β s β r? Loosely speaking, it is: Small variance of response propensities (in groups defined by x) Note that we know x s, x r and that we can manipulate x r by adaptive sampling (Schouten et al. 2013, Särndal & Lundquist 2014) 10/3/2015 NTTS Dan Hedlin, Stockholm University 7

8 Comparing three strategies 1. Perfect frame + random sample + unknown response propensities + we strive for response set balance 2. The same as 1 but with nonrandom sample 3. The same as 1 but with a restricted frame. There is auxiliary data on the frame. Only deficiency is undercoverage. E.g. large, good web panel. 10/3/2015 NTTS Dan Hedlin, Stockholm University 8

9 Scene Specify f(y X; β) for all variables Y for all N units. X for sampling design end estimation. 1. Analytic aim: inference about β 2. Descriptive aim: inference about Y s Some y used for post-stratification Y = Y post, Y m For robustness of post-stratification to nonresponse, see Särndal & Lundström (2005) We need f(y m s Y post s, X; β) for inference about Y s Further f(y m s Y post s, Z, X; β), Z indicates web panel membership in Strategy 3 10/3/2015 NTTS Dan Hedlin, Stockholm University 9

10 1. Sample selection ignorability criterion: f I s Y s, X = f I s Y post s, X True also for some nonrandom sampling designs, e.g. sample balanced designs (Little 1982, Smith 1983) 2. To be able to ignore nonresponse: f J r I s, Y s, X = f J r I s, Y post s, X 3. To be able to ignore web panel selection mechanism: f Y m s Y post s, Z, X; β = f Y m s Y post s, X; β (Little 1982, Smith 1983, Valliant et al. 2003) 10/3/2015 NTTS Dan Hedlin, Stockholm University 10

11 We restrict attention to ignorable sampling designs; nonrandom samples must be ignorable Hence it is Strategy 3 that is different. Does criterion 3 hold in practice? E.g. Sjöström (2012) found that sometimes it does, sometimes it does not. See also Baker et al. (2013). Note also that Strategy 3 has to a some limited extent always been in use in survey sampling, in particular in business surveys (cut-off sampling) 10/3/2015 NTTS Dan Hedlin, Stockholm University 11

12 Further issues Suppose you are successful in balancing the set of responses. Does it matter whether you have started from a random sample or a nonrandom, ignorable sample? It would seem that it does not. A more practical issue: If you strive for balancing the response set, is it easier to start from a random sample? What is best, balancing response set or adjusting through estimation? Some evidence that balancing is slightly better (Schouten et al. 2014) 10/3/2015 NTTS Dan Hedlin, Stockholm University 12

13 Of course, there is a broader picture (Schouten et al. 2012) 10/3/2015 NTTS Dan Hedlin, Stockholm University 13

14 References Baker, R. et al. (2013). Report on the AAPOR task force on non-probability sampling. American Association for Public Opinion Research. Holt, D. (2007). The Official Statistics Olympic Challenge: Wider, Deeper, Quicker, Better, Cheaper. (With discussion). The American Statistician, 61, Holt, D. (2008). Official statistics, public policy and public trust. Journal of the Royal Statistical Society, Series A, 171, Levins, R. (1966). The strategy of model building in population biology. American Scientist. Little, R. J.A. (1982). Models for Nonresponse in Sample Surveys. Journal of the American Statistical Association, 77, Särndal, C.-E. and Lundquist, P. (2014). Accuracy in Estimation with Nonresponse: A Function of Degree of Imbalance and Degree of Explanation. Journal of Survey Statistics and Methodology, Särndal, C.-E. and Lundström, S. (2012). Estimation in Surveys with Nonresponse. New York: Wiley. 10/3/2015 NTTS Dan Hedlin, Stockholm University 14

15 Schouten, B., Bethlehem, J., Beullens, K., Kleven, Ø., Loosvelt, G., Luiten, A., Rutar, K., Shlomo, N. and Skinner, C. (2012). Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators. International Statistical Review, 80, Shouten, B., Calinescu, M. and Luiten, A. (2013). Optimizing quality of response through adaptive survey designs. Survey Methodology, 39, Scouten, B., Cobben, F., Lundquist, P. and Wagner, J. (2014). Theoretical and Empirical Support for Adjustment of Nonresponse by design. Discussion paper, 2014/15, Statistics Netherlands. Sjöström, T. (2012). Självrekryterade jämfört med slumpmässigt rekryterade paneler. Novus, Sweden. (in Swedish) Smith, T.M.F. (1983). On the validity of inferences from non-random sample. Journal of the Royal Statistical Society, Series A, 146, Valliant, R., Dorfman, A.H. and Royall, R.M. (2000). Finite Population Sampling and Inference: A Prediction Approach. New York: Wiley. 10/3/2015 NTTS Dan Hedlin, Stockholm University 15