Methods for treating bias in ISTAT mixed mode social surveys

Transcription

1 Methods for treating bias in ISTAT mixed mode social surveys C. De Vitiis, A. Guandalini, F. Inglese and M.D. Terribili ITACOSM 2017 Bologna, 16th June 2017

2 Summary 1. The Mixed Mode in ISTAT social surveys 2. Theoretical framework 3. The survey context 4. The mode effect treatment for the CLT survey 5. Conclusions and open issues

3 1. The Mixed Mode in ISTAT social surveys Mixed mode is not a new way of data collection for ISTAT surveys but its treatment has been faced recently, from two points of view: Data collection Methodological In ISTAT several situations have occurred so far Mixed mode used primarily to address coverage issues of previously mono mode CATI surveys (-> web/cati) Mixed mode in longitudinal household surveys to reduce cost and burden (CAPI/CATI) Mixed mode used primarily to reduce survey cost whereas expanding population coverage, through the introduction of CAWI technique in traditionally PAPI surveys Multipurpose survey on households: Citizens and leisure time , CAWI/PAPI Multipurpose survey on households: Aspects of daily life : sequential CAWI/PAPI with a control mono mode sample PAPI

4 Why mixed modes? To contrast declining response rates and coverage, reducing also the total cost of the surveys. The use of different data collection techniques helps in contacting different types of respondents in the most suitable way for each of them so allowing a gain in population coverage and response rate Which drawbacks has this choice? The difficulty of control over mode effects and the confounding between selection and measurement effect 2. Theoretical framework Mode effect refers strictly to measurement error differences due to the mode of survey administration A selection effect generally occurs, due to the differences in the distributions of the respondents to the alternative modes, even if this is a desirable aspect of MM strategy How and when dealing with mode effect? Mainly in the planning of the survey (questionnaire and survey design) to limit measurement error as much as possible In the estimation phase to treat mainly the selection effect (possibly)

5 Issues and methods for treating mode effect 2. Theoretical framework To ensure the accuracy of inference in combining data collected with different techniques 1. Testing of the equivalence of measurement across modes assumption (Hox et al., 2015). 2. Disentangling selection and measurement effects through causal inference Methods based on Propensity Score approach (PS) (matching, subclassification, covariate adjustments) (Rosenbaum and Rubin, 1983; Rubin, 2006; D Agostino, 1998; Austin, 2011; Lee and Valliant, 2008). PS assumes mode selection ignorability, has the advantage of producing a weight adjustment, while involving the risk of finding unbalanced groups (Vandenplas et al., 2016). Methods based on Multiple Imputations (MI) (Rubin, 1987). MI allow to overcome the non-ignorability mode selection assumption (Park et al., 2016, Suzer-Gurtekin, et al., 2012), while correcting one by one the bias of target variables

6 2. Theoretical framework To treat mode effect the use of models is advisable and the availability of auxiliary variables is a crucial issue External sources: from registers or administrative data, sociodemographic and economic variables Survey variables Mode insensitive socio-demographic variables Mode preference (not yet introduced at ISTAT) Paradata (information about data collection phase)

7 3. The survey context The sample survey Citizens and leisure time Collects information about recreational and cultural activities in free time, such as sports, reading, cinema, music, the Internet, social relations, issues for the quality of life of people Based on a sample of about households, selected through a two stage sample design (municipalities/households) from the centralized municipal register (LAC) Mixed technique: sequential CAWI-PAPI A self-compiled questionnaire (CAWI) proposed in the inviting letter sent by ISTAT or, alternatively, direct interview with a questionnaire on paper, with an interviewer (PAPI)

8 3. The survey context To analyze mixed mode effects The selected sample of individuals was linked to an administrative data base (Archimede Project) through the fiscal code available from the selection frame (only for the individuals belonging to the selected households at time of selection) The linkage gave a good result (95,4% of linked units), uniformly distributed between CAWI and PAPI, but not between respondent and not respondent (3% of respondents do not have FC as there are additional members of de facto households, not listed in the selection frame but included in the sample)

9 Table 1. Response, linkage and selection rates for CLT survey after the linkage to Administrative DB Response and mode choice LINKED LINKAGE Total NOT LINKED % % % NOT RESPONDENT ,7% ,8% ,7% RESPONDENT ,3% ,2% ,3% CAWI ,0% ,7% ,9% PAPI ,0% ,3% ,1% Total % %

10 Figure 1. The informative situation for CLT survey after the linkage of the selected sample to the Administrative DB Linkage Response Mode Y X 1 X p LINKED NOT LINKED RESPONDENT NONRESPONDENT RESPONDENT NONRESPONDENT CAWI O PAPI CAWI PAPI O O O O O O O O

11 4. The mode effect treatment for the CLT survey For the treatment of mode effects in the Citizen and leisure time survey, we are experimenting different methods: A. the Propensity Score Subclassification method (Rosenbaum and Rubin, 1983; Austin, 2011; Vandenplas, 2016) assuming mode selection ignorability B. the Parametric Fractional Imputation (PFI) method (Park et al., 2016) Allowing to consider also nonignorability of mode selection the PAPI is taken as a reference survey mode the imputation produces counterfactual values for the survey variable y as if the CAWI respondent had responded with the PAPI technique The ultimate goal of the analysis is the comparison of the estimates of some target parameters obtained using alternative approaches based on different assumptions, with the aim to obtain an assessment of these assumptions as well

12 4. The mode effect treatment for the CLT survey Given the operational context, the choice of the methods to be implemented for the analysis and treatment of mode-effects has to deal with some issues: missing values on covariates, so that method for incomplete data analysis should be considered (Ibrahim et al., 2005), otherwise we will have to face the issue of extending the results to the incomplete part of the sample the total nonresponse which results downstream of the selection process and can not be addressed with usual methods For the moment, assuming the MAR hypothesis, we concentrate our analysis to the set of respondent with a complete set of a reduced number of covariates X Besides we leave the total nonresponse issue to a further phase of the study

13 A. The application of the Propensity Score Subclassification 4. The mode effect treatment for the CLT survey The purpose of the implementation of the Propensity Score Subclassification method is to correct the selection effect through the calculus of weights which adjust the distribution of the CAWI and PAPI respondents Propensity score model is defined at household level, as the choice of the survey mode depends on household; P M = CAWI X for the case of ignorability, is a binomial logistic model at household level Survey mode ~ geo area + municipal type + household type + household income class + higher education level + occupation type + citizenship Household type: one-component under 55, one-component over 54, couple with children at least one under 25, couple with children without under 25, couple without children, one parent at least one under 25, one parent without under 25, other types Higher education level: below/equal/above high school diploma Occupation type: Prevalence of: employed, self employed, not in labor age, mixed types Municipal type: Metropolitan cities, metrop. area, other munic. <2000, , , >50000 Income class: 5 quintiles ( , , , ) Citizenship: Italian/Foreign household

14 4. The mode effect treatment for the CLT survey The application of the Propensity Score Subclassification : Propensity Score Subclassification steps: estimation of the propensity score model (choice model) parameters definition of strata of respondents (CAWI and PAPI) based on quintiles and deciles of the propensity score distribution validation of the balancing assumption in each stratum (independence of all X from the mode) for each balanced group k, the calculus of weighs that equate the weighted proportion of CAWI respondent households with the proportion of PAPI respondent households in the same stratum w k = Τn papi n k,web Τn web n k,papi

15 The application of the Propensity Score Subclassification 4. The mode effect treatment for the CLT survey An evaluation of measurement error within the balanced groups is obtained through the test of the independence between Y and the mode d k,web = F k,web F k,web n k,web, d k,papi = F k,papi F k,papi n k,papi A global evaluation of mode effects using the weights in the balanced groups (Vandenplas, 2016) selection effect, by the difference between weighted and not weighted estimates on web respondents S web y = σ n web i=1 y i,web σ n web i=1 w k,i y i,web n web n web mode effect, measured by the differences between weighted web and not weighted PAPI estimates M web y = σ n web i=1 w k,i y i,web n web σ n papi y i=1 i,papi n papi

16 Table 2. Estimates of the parameters of mode choice model (accuracy 71.9% ) Covariates Estimate Std. Error Pr > ChiQuadr (Intercept) <.0001 *** North-West <.0001 *** North-East <.0001 *** Central Italy <.0001 *** South Italy <.0001 *** Metropolitan cities <.0001 *** Metropolitan area municipalities ** Other >2.000 inhabitants ** Other inhabitants <.0001 *** Other inhabitants One-component under ** Couple with children at least one under Couple with children without under Couple without children <.0001 *** One parent at least one under * One component over Other types ** Below HS diploma <.0001 *** HS diploma < <.0001 *** *** <.0001 *** Mixed citizenship household italian <.0001 *** Prevalence of Employed *** Prevalence of Self-Employed Prevalence of Non labor age <.0001 *** Signif. codes: 0 *** ** 0.01 * REF: Isles, Other municip. > inhab., One parent with no one under 25, Above HS diploma, >46.119, Foreign househ., Mixed labor type

17 Results of Propensity score subclassification The balancing condition is verified in all 10 deciles Figure 1. Measurement effect within groups for some target variables Tv habit No Everyday Sometimes No Internet access Yes In the last 3 months In the last month PC use From 3 months to a year ago Sport More than a year ago 2 to 6 From 6 More than a months ago months to a year ago year ago Never Never CAWI relative difference PAPI relative difference

18 Results of Propensity score subclassification Table 3. Selection and measurement effect for some target variables Target variable Web mean Weighted Web mean PAPI mean Selection effect Measurement effect TV HABIT No 0,136 0,113 0,090 0,023 0,024 Everyday 0,693 0,666 0,725 0,028-0,059 Sometimes 0,163 0,144 0,165 0,019-0,021 PC USE In the last 3 months 0,698 0,537 0,497 0,161 0,040 From 3 months to a year ago 0,028 0,030 0,029-0,003 0,001 More than a year ago 0,054 0,056 0,045-0,002 0,011 Never 0,213 0,299 0,401-0,087-0,101 INTERNET ACCESS (household) No 0,152 0,226 0,368-0,074-0,142 Yes 0,848 0,703 0,632 0,145 0,071 SPORT In the last month 0,413 0,306 0,252 0,107 0,054 2 to 6 months ago 0,072 0,061 0,057 0,012 0,004 From 6 months to a year ago 0,028 0,023 0,022 0,005 0,001 More than a year ago 0,219 0,189 0,211 0,030-0,023 Never 0,268 0,352 0,458-0,084-0,106

19 4. The mode effect treatment for the CLT survey Further analyses for the use of Propensity Score Classification The sampling variability has to be taken into account by means of a simulation to assess significance of mode effects Further experimentations are needed: Sensitivity analysis of the choice model Issues to be faced and solved in order to define a correction factor for all the respondents in the sample: The incompleteness of covariates (around 8% of total respondents)

20 4. The mode effect treatment for the CLT survey B. The Parametric Fractional Imputation (PFI) method The PFI addresses in a comprehensive way all the components of the mode effect evaluation, including the nonignorability assumption Following the notation in Park et al. (2016), we started to define structural model f(y X; θ) the measurement error model g(y CAWI y; α), assuming PAPI as reference mode the choice model P(M=CAWI X, y) We consider two dichotomous survey variables: internet access (1=yes, 0=no) sport activity in the last month (named sport activity, 1=yes, 0=no)

21 4. The mode effect treatment for the CLT survey The PFI implementation : Structural models f for the dicotomic independent variable y is a binomial logistic model at individual level, on PAPI respondents Model 1 y ~ region + sex + age class + marital status + education + citizenship + occupation type + income class + income source Measurement model g on y (Model 2) is a linear model defined at individual level. In the model g, the covariate is the predicted value y estimated from the structural model f Choice model is defined at household level, P M = CAWI X, y for the case of nonignorability Survey mode ~ region + # components + household income + # components by sex + # components by age groups + # components by marital status + # components by education + y

22 4. The mode effect treatment for the CLT survey The PFI implementation : Structural Model 1 and Measurement Model 2 are used together with choice Model 3 to define the imputation model f(y PAPI X, y CAWI ) for the PFI method. The imputation model, obtained through the Bayes formula, predicts imputed values (unobserved response PAPI) for the CAWI respondents. The parameters of the models and the predicted values have to be estimated via an iterative algorithm. In each iteration several imputed values, with their fractional weights, are created for each CAWI respondent. Each fractional weight is computed as the expected value of the imputation model. The purpose is deriving an unbiased estimate of the interest parameter using the conditional expectation of the imputed values for the CAWI respondents. The fractional imputation requires ad hoc algorithms and is computationally heavy Our intent is to complete both the applications to evaluate, through the PFI on the main survey variables, the ignorability assumption and the size of measurement error, in order to correct, in case of good results, the estimates only for the selection effect through the propensity score weighting

23 6. Final considerations and future developments The current effort for the evaluation of mode effects aims at providing useful indications for future survey planning (better instruments, mode insensitive covariates, mode preference.) as in general measurement error should be prevented while selection effect only should be treated in the estimation phase, using well fitting models ISTAT is planning a whole redesign of the social surveys in the context of the permanent census based on the master sample, for which mode preference and benchmark covariates will be collected by a one-mode face-to-face survey We expect that some process issues, that presently limit the quality of the auxiliary variables and the linking to the survey sample units, will improve in the future

24 Thank you for your attention!