On the Importance of Representative Online Survey Data : Probability Sampling and Non-Internet Households

Transcription

1 On the Importance of Representative Online Survey Data : Probability Sampling and Non-Internet Households Annelies G. Blom Principal Investigator of the German Internet Panel Juniorprofessor for Methods of Empirical Social Research, in particular Internet Panel Surveys University of Mannheim, Germany

2 Online Panel Research Surge in commercial online panels in recent years Mostly non-probability panels where panelists self-select themselves into the panel None include the offline population Increasing doubts in the representativeness of non-probability online panels E.g British inquiry into election polls based on online panels See also Yeager et al. (2011) Public Opinion Quarterly. 2

3 Online Panel Research To counter these problems, a handful of online panels decided to conduct recruitment via strict probability methods (offline recruitment via standard face-to-face or telephone probability sampling methods) include the offline population (providing the necessary equipment to offline sample members or interviewing offliners in a different mode) 3

4 German Internet Panel (GIP) Germany s first probability-based online panel of the general population Bi-monthly longitudinal data on political attitudes and preferences (e.g. left-right placement, economic risk aversion, preferences for redistribution) Area-based sampling with recruitment via face-toface interviews Recruitment of non-internet households (equipment with internet and simple computers) 4

5 Offliner equipment BenPC (2012) GiWi (2014) 5

6 Central research question Does the recruitment of offline households positively contribute to the sample representativeness of the German Internet Panel? 6

7 Research questions What is the recruitment rate among offliners and onliners? How different are offliners from onliners? Which characteristics explain participation in the online panel? How does the inclusion of offliners influence the representativeness of the sample? 7

8 What is the recruitment rate among offliners and onliners? 8

9 Recruitment rates 9

10 How different are offliners from onliners? 10

11 Onliners are younger more likely to be male living in larger households living in urban areas higher educated more politically interested than offliners 11

12 Who is online? 12

13 Which characteristics explain participation in the online panel? 13

14 Participation more likely for younger sample members onliners small households people who were personally interviewed during the face-to-face recruitment 14

15 Who participates? 15

16 How does the inclusion of offliners influence the representativeness of the sample? 16

17 Impact of offliners on sample representativeness We compare four (sub-)samples of the GIP to known population distributions Age Gender Education Household size (Sub-)samples investigated All onliners All offliners Total GIP sample 17

18 Representativeness: Age 18

19 Representativeness: Age 19

20 Representativeness: Age 20

21 Representativeness: Education 21

22 Representativeness: Household size 22

23 Representativeness: Gender 23

24 Take home messages regarding representativeness Do offliners differ from onliners? Yes, onliners and offliners differ significantly from each other. How does being offline affect participation in the GIP? Offliners participate at significantly lower rates than onliners. How does the inclusion of offliners influence the representativeness of the sample? With respect to age, education and household size it results in a more representative sample. 24

25 Future perspective: Internationalisation 25

26 Probability-based online panels First probability-based online panels, which include offline households: FRISPP in the United States (2007) Knowledge Networks panel in the United States (2007) LISS panel in the Netherlands (2007) Further academically-led probability-based online panels, which include offline households: GIP in Germany (2012) ELIPSS Panel in France (2012) GESIS Panel in Germany (2014) First joint wave of LISS, ELIPSS, GIP and GESIS Panel in May/July 2014 Since early 2014 academically-led probability-based online panels from Northern Europe joined the initiative: Norwegian Citizen Panel (NCP) Swedish Citizen Panel (SCP) SSRI online panel in Iceland 26

27 Retention: GIP, LISS, ELIPSS and GESIS Panel 27

28 Cross-national collaboration Coordinated joint waves LISS, ELIPSS, GIP, GESIS Panel: Scoping study Pre- and post-election panel on the EU parliamentary elections 2015 LISS, ELIPSS, GIP, GESIS Panel: Investigating reform preferences cross-nationally 2016 LISS, GIP, SCP, SSRI: Longitudinal tracking of reform preferences cross-nationally Cross-country replication through open calls LISS, ELIPSS, GIP, NCP and SCP Participation in small-scale data collections across many countries, for example, the Multi-National Study of Questionnaire Design (MSQD) which was conducted across 15 countries. European goal: European Internet Panel Study (EIPS) 28

29 Concluding remarks Probability-based online panels, in particular when they include the offline population, still tread on unexplored grounds. There are a few research projects into their quality, one of them I have presented today. Such projects typically find that: the representativeness of nonprobability panels is unreliable (sometimes they are representative sometimes they are not). the representativeness of probability online panels is better and - most importantly - more reliably so. including offliners in probability online panels adds to the sample representativeness (contribution to correlations less clear). 29

30 References for this presentation Blom, A.G., J.M.E. Herzing, C. Cornesse, J.W. Sakshaug, U. Krieger, and D. Bossert. (2016) Does the Recruitment of Offline Households Increase the Sample Representativeness of Probability-Based Online Panels? Evidence From the German Internet Panel, Social Science Computer Review. Online First on 2 nd June 2016, doi: / Blom, A.G., M. Bosnjak, A. Cornilleau, A.-S. Cousteaux, M. Das, S. Douhou and U. Krieger (2016). A Comparison of Four Probability-Based Online and Mixed- Mode Panels in Europe, Social Science Computer Review, 34(1), 8-25 Address of correspondence: blom@uni-mannheim.de 30

31 Additional references Blom et al. (2015). Setting up an online panel representative of the general population: The German Internet Panel. Field Methods, 27, Eckman (2015). Does the inclusion of non-internet households in a web panel reduce coverage bias? Social Science Computer Review, 34, Erens et al. (2014). Nonprobability web surveys to measure sexual behaviors and attitudes in the general population: A comparison with a probability sample interview survey. Journal of Medical Internet Research, 16, e276. Leenheer & Scherpenzeel (2013). Does it pay off to include non-internet households in an internet panel? International Journal of Internet Science, 8, Mohorko et al. (2013). Internet coverage and coverage bias in Europe: Developments across countries and over time. Journal of Official Statistics, 29, 1 15 Revilla et al. (2016). What is the gain in a probability-based online panel of providing internet access to sampling units who previously had no access? Social Science Computer Review, 34, Rookey et al. (2008). Does a probability-based household panel benefit from assignment to postal response as an alternative to internet-only? Public Opinion Quarterly, 72, Schonlau et al. (2009). Selection bias in web surveys and the use of propensity scores. Sociological Methods and Research, 37, Yeager et al. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly, 75,

32