General Online Research Conference (GOR 17) March 2017 HTW Berlin University of Applied Sciences Berlin, Germany

Transcription

1 General Online Research Conference (GOR 17) March 2017 HTW Berlin University of Applied Sciences Berlin, Germany Edith de Leeuw, Utrecht University Joop Hox, Utrecht University Benjamin Rosche, Utrecht University Predicting Nonresponse and Attrition in a Probability-Based Online Panel Contact: Edith de Leeuw e.d.deleeuw@uu.nl Suggested citation: de Leeuw, Edith, Hox, Joop, Rosche, Benjamin Predicting Nonresponse and Attrition in a Probability- Based Online Panel General Online Research (GOR) Conference, Berlin. This work is licensed under a Creative Commons Attribution 4.0 International License (

2 Predicting Nonresponse and Attrition in a Probability-Based Online Panel Edith de Leeuw, Joop Hox, & Benjamin Rosche Department of Methodology & Statistics, Utrecht University General Online Research Conference Berlin, March 2017

3 Online Panels Online Panels one of the fastest growing data collection tools Probability-based and Non-probability based Probability-based state of the art Europe and USA Nonresponse and attrition always threat Also to probability-based panels However, advantages probability based panels are Know and can calculate response rates Know who is not responding, attriting Question is: can we do something about it? 3

4 Research Question Theories on Nonresponse Social Exchange Theory Planned Behaviour (Reasoned Action) Social Psychological (special case: Leverage saliency) Important variables Socio-Biographical indicators & Attitudes Do survey attitudes predict wave nonresponse and attrition? Do survey attitudes predict better than standard indicators (e.g., age, education, income, urbanization)?

5 Available Data Dutch LISS Panel ( Probability-based online panel since October 2007 Core questionnaire every year (longitudinal part) Varying questionnaires every month (cross-sectional) Survey Attitude Scale in core ( ) Nine item scale (de Leeuw et al, 2010) 34 variables (socio-demographic, psychographic) Based on expert ratings selection: 13 clear indicators of nonresponse, used as covariates Dependent variable based on individual panel member s Number of completed interviews per year Number of invitations per year Period

6 Survey Attitude Scale 3 subscales: enjoy, value, burden 3 items/scale, reliabilities ok Nice simple factor structure

7 Survey Attitude Specific Trait Enjoy Value Burden State at time t Common Trait Is survey attitude a stable individual characteristic or does it fluctuate over time? State and Trait Model (Bons et al, 2014) Variance 37-62% trait, 6-33% state Use trait & state variables to predict nonresponse = mean score & deviation over time

8 Analysis Strategy Multilevel Negative Binomial Regression (Respons) Dependent variable: count of completes per year Offset: count of invites per year Period Step1: only survey attitude predictors Step2: all covariates (nonresponse correlates) added Period Cross-validation of prediction models step 1 and 2 Completes and invites Survival Analysis on Attrition Attrited between

9 Results 1 Dependent variable: Number of completed interviews p.a. Model 1: Survey Attitude Model 2: SAS + covariates Exp(B) Exp(B) Intercept ** Year (2008 = 0) 0.963** 0.964** Survey attitude scale Enjoyment: mean 1.218** 1.201** Enjoyment: deviation 1.021** 1.021** Value: mean 1.084** 1.070** Value: deviation Burden: mean 0.883** 0.892** Burden: deviation 0.992** 0.992** Female 1.031* Age 1.006** Years of education 0.994** Migrant Dwelling: Self-owned S Urbanization Household income Household size 0.989* SimPC 0.959* Social (generalized) trust Voted Dissatisfied leisure time 0.994** Agreeableness 0.974** Predicting Response 6 Survey Attitude predictors predict 62.8% of total explained variance All trait attitudes significant 2 of 3 state attitudes significant: enjoyment & burden Adding covariates does not change attitude model

10 Results 2 Correlation Predicted response rate using model 1 (survey attitude scale, only trait-part) Predicted response rate using model 2 (adding covariates) SAS trait based on scores of: Observed 2014 R=0.120 R=0.140 R=0.238 R=0.304 R=0.295 R=0.335 response rate Observed 2015 response rate R=0.108 R=0.135 R=0.230 R=0.295 R=0.292 R=0.330 Cross-validation using holdout sample Predicting response in from the model for Better prediction with more information More trait scores and closer in time Covariates

11 Results 3 Dependent variable: Dropout Model 1: survey attitudes Model 2: add covariates Coef. Coef. Intercept 0.920* Year (2008=0) ** *** Survey attitude scale Enjoyment: mean ** *** Enjoyment: deviation Value: mean ** *** Value: deviation * *** Burden: mean 0.271** 0.261*** Burden: deviation Female Age Years of education *** Migrant Dwelling: Self-owned Urbanization Household income 0.000*** Household size SimPC * Social (generalized) trust Voted *** Dissatisfied leasure time Agreeableness 0.248* Survival Analysis: predicting Dropout All Trait scores significant, only 1 State Survey Attitudes predict 17.04% of the observed dropout Adding 13 Covariates increases this to 17.22%

12 Summary Similar results from Multilevel Negative Binomial Regression predicting counts of responses per year and survival analysis predicting panel dropout Survey Attitude Trait (mean) score predicts well Survey Attitude State (deviation) score predicts less Usual demographics important but do not diminish importance of attitudes Explained variance attitudes around 15-20% With demographics added increased to 25-30% Prediction on later years better if more and more recent measures of predictors are used

13 Conclusion Traits are presumably difficult to change States are temporarily, can be influenced Enjoyment most important, Burden second For Enjoyment and Burden subscales both trait and state predict, for Burden only trait Stress that survey is enjoyable and easy to fill in Make surveys enjoyable and easy (cf. Dillman) Gamification? (cf. Cape, Keusch&Zhang, Puleston) Short ( bonsai ) surveys? (cf. Puleston)

14 Acknowledgements We thank Annette Scherpenzeel, Corrie Vis & Miquelle Marchand (LISS-CentERdata) for their knowledgeable assistance in procuring the LISS data We thank 31 international experts in survey methodology and nonresponse, who rated theoretical indicators of nonresponse on importance. We are very grateful for your labour of love!

15 References Cape, P. (2016) Gammifying questions using tekst alone. GOR 2016-archive Dillman, D.A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley (and later works e.g. Dillman, D.A., Smyth, J.D. & Christian, L.M. (2009) Internet, Mail and Mixed-mode surveys: The Tailored Design Method. New York: Wiley). Bons, H., Hox, J., de Leeuw, E & Schouten, B. (2015). Stability of the survey attitude scale over time: A latent state-trait analysis. Poster presented at the JOS 30 th anniversary conference, SCB, Stockholm. De Leeuw, E. D., Hox, J. J., Lugtig, P., Scherpenzeel, C. V., Goritz, A., & Bartsch, S. (2010). Measuring and Comparing Survey Attitude Among New and Repeat Respondents Cross-Culturally. Wapor 63 annual conference. Chicago. Keusch, F. & Zhang, Ch. (2015) A review of issues in gammified surveys, SSCR, Puleston J. (2015). The art of asking questions. Workshop at the 2015 GORconference, Cologne Puleston J. (2012) Gammification 101- from theory to practice, part 1 & 2. Quirk s Marketing Research Media

16 Appendix A Survey Attitude Scale Nine items, based on literature Analysis showed that 3 constructs existed and were measured reliably Enjoyment, Value, & Burden (de Leeuw et al, 2010) These are described in the following slides In parenthesis reference to literature where these questions were used earlier All questions were to be answered on 7-point scale, ranging from Totally Disagree to Totally Agree. The response scales were endpoints labeled only.

17 Survey Attitude Scale Three Constructs: I Survey Enjoyment I really enjoy responding to questionnaires through the mail or Internet (Cialdini/Rogelberg) I really enjoy being interviewed for a survey (Cialdini/Rogelberg) Surveys are interesting in themselves (Stocké)

18 Survey Attitude Scale Three Constructs: II Survey Value Surveys are important for society (Stocké) A lot can be learned from information collected through surveys (Rogelberg) Completing surveys is a waste of time (-) (Rogelberg/Singer)

19 Three Constructs continued Three Constructs: II Survey Burden I receive far to many request to participate in surveys (Cialdini) Opinion polls are an invasion of privacy (Goyder) It is exhaustive to answer so many questions in a survey (Stocké)

20 Appendix B Indicators of Nonresponse as Covariates Gender Age Years of education Migrant or not Household size Household income Type of dwelling Urbanization SIMPC Generalized (Social)Trust Voted in at least one national election Opportunity costs (dissatisfaction with amount of leisure time) Agreeableness (big five)