www.ssoar.info The effect of the refusal avoidance training experiment on final disposition codes in the German ESS-2 Schnell, Rainer; Trappmann, Mark Veröffentlichungsversion / Published Version Arbeitspapier / working paper Zur Verfügung gestellt in Kooperation mit / provided in cooperation with: SSG Sozialwissenschaften, USB Köln Empfohlene Zitierung / Suggested Citation: Schnell, Rainer ; Trappmann, Mark ; Universität Konstanz, Center for Quantitative Methods and Survey Research (CMS) (Ed.): The effect of the refusal avoidance training experiment on final disposition codes in the German ESS-2. Konstanz, 2006 (Arbeitspapier / Universität Konstanz, Zentrum für Quantitative Methoden und Surveyforschung 3/2006). URN: http://nbn-resolving.de/urn:nbn:de:0168-ssoar-114189 Nutzungsbedingungen: Dieser Text wird unter einer Deposit-Lizenz (Keine Weiterverbreitung - keine Bearbeitung) zur Verfügung gestellt. Gewährt wird ein nicht exklusives, nicht übertragbares, persönliches und beschränktes Recht auf Nutzung dieses Dokuments. Dieses Dokument ist ausschließlich für den persönlichen, nicht-kommerziellen Gebrauch bestimmt. Auf sämtlichen Kopien dieses Dokuments müssen alle Urheberrechtshinweise und sonstigen Hinweise auf gesetzlichen Schutz beibehalten werden. Sie dürfen dieses Dokument nicht in irgendeiner Weise abändern, noch dürfen Sie dieses Dokument für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, aufführen, vertreiben oder anderweitig nutzen. Mit der Verwendung dieses Dokuments erkennen Sie die Nutzungsbedingungen an. Terms of use: This document is made available under Deposit Licence (No Redistribution - no modifications). We grant a non-exclusive, nontransferable, individual and limited right to using this document. This document is solely intended for your personal, noncommercial use. All of the copies of this documents must retain all copyright information and other information regarding legal protection. You are not allowed to alter this document in any way, to copy it for public or commercial purposes, to exhibit the document in public, to perform, distribute or otherwise use the document in public. By using this particular document, you accept the above-stated conditions of use.
Working Paper 3/2006 The Effect of the Refusal Avoidance Training Experiment on Final Disposition Codes in the German ESS-2 Rainer Schnell/Mark Trappman March 22, 2006 Center for Quantitative Methods and Survey Research University of Konstanz, Germany
Abstract The implementation of a Refusal Avoidance Training (RAT) within wave 2 of the German part of the European Social Survey (ESS) successful reduced the amount of reported refusal by nearly 7%. The effect of the reduction was compensated by a nearly equal increase in the proportion of non-contacted designated respondents. This effect may be due to non-random allocation of trained interviewers. Further randomized experiments are neccessary to separate the effects of RAT on response rates.
1 Refusal Avoidance Training (RAT) Based on Groves/McGonagle (2001) and interviewer manuals of the Survey Research Center instructional materials of a German Refusal Avoidance Training (RAT) had been developed between 2003-2005. The goal of a RAT is reducing refusal rates by teaching interviewers how to deal with doorstep interactions by maintaining interaction with the respondents and tailoring the arguments to them. The RAT-Training was developed for in person teaching of face-to-face interviewers and as a multimedia selfteaching unit for CATI-interviewers. 1 1.1 Refusal Avoidance Training in the German ESS Since data on the effect of RATs on interviewer fieldwork is rare, the project group tried to conduct a RAT implementation under realistic field work conditions. The German ESS-group kindly allowed a test of the RAT training within the German-ESS-Wave- 2. 2 Due to the restrictions of ongoing fieldwork within a commercial fieldwork agency, the project group had no influence on the sampling of interviewers, their workload or working conditions. Nevertheless, 28 already experienced interviewers received a RAT in addition to their basic training. 1.2 Implementation details of the RAT The training took place at the training facilities of the fieldwork agency in Bonn as a one day personal education of the interviewers on 2004/08/26. The coach was a professional coach without special experience in survey methodology, but was trained and assisted by a member of the research group. In addition to the coach, her assistant and the 28 interviewers (of whom only 26 actually conducted interviews), two representatives of the German ESS-Group and representatives of the fieldwork agency were present. The training took two hours. At first, the interviewers collected and organized arguments given by reluctant respondents. In this plenary session suitable responses to these arguments were discussed. During the next session groups of 2-3 interviewers tried to find arguments to respondent objections. Finally, the written interviewer responses were collected by the coach and discussed in a plenary session. 1 This work was part a project granted to the first author by the German Research Foundation (DFG). 2 We would like to thank the German ESS-group and the fieldwork agency Infas (Bonn) for their kind permission to conduct the experiment within the ESS wave 2. 3
2 Effect of RAT on Final Disposition Codes The data set contains information on 15657 contacts to 5883 designated respondents by 207 interviewers. During fieldwork, 5868 addresses were contacted, resulting in 2913 interviews (49.6%). Considering only the last contacts, which were done face-to face, data on 5086 respondents are available. Overall, 2913 (57.3%) of these were interviewed. The difference in response rate between RAT-trained and standard interviewers (59.3% vs. 56.8%) is significant only on a 10% level and a one-sided test (p=0.0855). A plot of proportions of final disposition codes by interviewer group (see 2.1) reveals only two remarkable differences between RAT and standard interviewers: The RAT-trained interviewers have a 7.2% lower refusal rate 1 than the standard interviewers (p < 0.0001). But this advantage of the RAT-group is compensated by a higher rate of non-contacts (5.7% difference, p < 0.0001). This strange result deserves a more detailed analysis. 2.1 Effects of alternating trained and untrained interviewers on the probability of a successful interview In order to examine the effect of RAT on the probability of a successful interview, alternating sequences of trained and untrained interviewers working on one designated respondents were examined. 35 designated respondents generated contact histories with trained and untrained interviewers. In 21 sequences an untrained interviewer was followed by a RAT-interviewer (U-RAT), in 14 sequences a RAT-interviewer was followed by an untrained interviewer (RAT-U). 13 of the 21 U-RAT-sequences resulted in an interview (61.9%). Of the 14 RAT-U-sequences, 13 (92.9%) resulted in an interview. Of course, despite the overlapping confidence intervals, this yields a significant test of equal proportions (z=2.05, p=0.04). The untrained interviewers seem to be more successful than the RAT interviewers. Since the allocation of cases to interviewers was not random and this result is based only on 9 different interviewers, this conclusion is not definitive. 1 Refusal is defined as an explicit decline of cooperation by a designated respondent or a household member. If we consider an extended definition of refusal, the difference between RAT and standard interviewers is 7.1%. The extended definition includes reported gatekeeper problems, missed appointments and appointments after the agency ceased fieldwork. 4
Refusal-E Refusal Other Nocontact Interview Illness Frame 0.0 0.1 0.2 0.3 0.4 0.5 0.6 proportion Training Standard RAT Figure 2.1: Final Disposition Codes by type of training 2.2 Effects of trained and untrained interviewers on refusal conversion Excluding all cases with interviewer changes, only sequences with designated respondents who at least during one contact refused to participate, were analyzed to examine the effect of RAT on refusal conversion. The difference in refusal rates between RAT-trained and non-trained interviewers (19.8% vs. 22.3%) is not significant. 2.3 Possible explanations of the compensating effects Since the goal of the training (reduction of refusal) seems to have been achieved, the compensating effect of increased non-contact is puzzling. Since this has been a nonrandomized experiment, rival hypotheses come to mind. For example, there could be differences in the allocation of areas with hard-to-contact respondents or differences in workload or duration or time of fieldwork. These obvious variables were examined by bivariate comparisons and a multilevel model containing all control variables. 2.3.1 Effects of urban areas RAT-trained interviewers are working slightly more often in urban areas. Interestingly, the number of inhabitants in the town of residence of designated respondents causes an interaction between response rate and interviewer training. In a classification of 5
10 town size classes, the percentage of completed interviews is higher for the RATinterviewers in the towns with fewer inhabitants (57.7% vs. 50.4%, classes 1-7, z=2.56, p=0.01) and lower for the larger towns (46.4% vs. 50.4%, classes 8-10, z=1.87, not significant). A similar interaction effect can be seen in the non-contact rates: In small towns, the percentage of non-contacted respondents is similar for the interviewer groups (RAT-interviewers 10.0%, standard-interviewers 9.3%, not significant), but in larger towns the non-contact rates reverse: RAT 16.3% vs. standard 6.5% (z=7.88, p <0.001). Remarkably, there is no reversal in effects on refusal rates according to town size: The refusal rates differ between RAT-interviewers and standard interviewers 15.6% vs 25.7% (z=3.74, p=.0002) in small towns and 22.6% vs. 27.3% (z=2.44, p=.015) in large towns. 2.3.2 Effects of federal states Only in 11 federal states both kind of interviewers conducted fieldwork. In 9 of these 11 states, RAT-interviewers showed lower refusal rates. On the other side, in 8 of 11 states the RAT-interviewers show higher non-contact rates. Overall, the response rates are higher for RAT-interviewers in 6 states and lower in 5 states. 2.3.3 Duration and date of fieldwork The duration of fieldwork is longer for RAT-interviewers than for standard interviewers (92 vs. 60 days). This is due to an earlier start (mean: 2004/10/10) with the mean end-time being approximately the same for interviews both groups. Interestingly, the duration between first and last contact to a designated respondent is shorter for the RAT-interviewers than for standard interviewers (23 vs. 27 days, t=3.69, p=.0002). Even more remarkable, this difference is even larger for respondents which were finally non-contacted (22 vs. 32 days, t=3.12, p=.002), since in these cases better interviewers should increase their efforts. In order to test the hypothesis, that the failure of RAT-interviewers was due to different school holidays in different federal states, in an additional analysis three groups of designated respondents we analyzed separately: Respondents whose first contact occurred before the first school holidays, respondents whose first contact occurred within school holidays (2004/10/04-2004/10/29) and respondents whose contacts occurred after the end of the school holidays. Only in the first group the non-contact rates of RATinterviewers were comparable to those of standard interviewers, in the other two groups, the non-contact rates were much higher for RAT-interviewers. 2.3.4 Workload Mean workload for RAT-interviewers is much higher than for non-rat interviewers (41.2 vs. 27.8, t=3.14, p=.002). The boxplot in 2.2 shows that this difference is not due to some outliers. Especially Q25 and Q75 are much higher for RAT-interviewers. This may indicate that RAT-interviewers were used for additional (and probably more difficult) cases by the end of field time. 6
non RAT RAT workload 0 20 40 60 80 100 Graphs by interviewer training Figure 2.2: Workload by type of training 2.4 A multilevel model of final disposition codes Since the 26 RAT-interviews generated 2979 contacts, the contacts can not be considered independent. Multilevel models may be used for a multivariate model respecting these dependencies (within an interviewer) of the contact results. Due to the fact, that in many primary sampling units only one interviewer conducted fieldwork, the effects of sampling points and interviewers can not be separated. The effects of sampling points were therefore modeled by using the measure of town size as an interviewer characteristic. 1. For each of the most important fieldwork results (non-contact, refusal, interview) a separate random-intercept logit model was computed. Interviewer workload, duration of fieldwork, urban vs non-urban area, interviewer training and the interaction of urban vs non-urban area with interviewer training were used as independent variables. 2 In the interview model, none of the independent variables is significant. In the noncontact model, no effect of RAT-training within non-urban areas can be found. But in urban areas, the log-odds (1.23) of the RAT-training is significant. This significant interaction can not be found in the refusal model, but nevertheless the log-odds of the RAT-training (-0.58) is significant in non-urban areas and not-significant in urban areas (-0.20). In sum, the results of the multilevel models confirm the previous bivariate findings. 1 In a few cases, interviewers work in different classes of town size. In this case, the mode of town size of these interviewers were used. 2 Since the effect of duration of fieldwork can not be clearly separated from the dependent variable (endogeneity), we will not interpret the coefficients of this variable. 7
2.5 Analysis of field reports Finally, the number of entries and the number of different codes in the field reports will be examined. 2.5.1 Number of contacts There is no significant difference between the training groups in the number of contact attempts. This is also true just for finally not interviewed designated respondents: The RAT-group attempted 2.39 contacts, the standard interviewers 2.50. But among the finally not-contacted designated respondents, the RAT-group attempted 2.51 contacts, the standard interviewers 3.30 contacts. The available data is not sufficient to examine the causes of this significant difference (t=3.38, p <.001) further. 2.5.2 Number of different field codes As a proxy for the attention paid by interviewers to document the fieldwork, the number of different result codes per interviewer was analyzed. A low number of different codes may be due the time saving method for an interviewer to fill out contact records monotonously by using the same codes over and over again. Empirically, this is wrong: Both interviewer groups use about 2.1 different codes per target person. So we have no evidence that the difference in reported fieldwork is due to differences in reporting accuracy. non RAT RAT proportion of non-contacts 0.1.2.3.4.5 Graphs by interviewer training Figure 2.3: No-contact rates by interviewer training type 8
2.6 Outliers Since the number of interviewers (n=26) is small, outliers may be of special importance. But no hints for outliers could be found. For example, the box plot for non-contact rates by interviewer training shows that the median of RAT-interviewers is higher than the the 75-Percentile of the standard-interviewers. So the effect of higher non-contact rates for RAT-interviewers is not due to a small number of outliers among the interviewers. 2.7 Summary RAT seems to be successful in reducing refusal rates: The RAT-trained interviewers achieved a significantly lower amount of refusal. However, this difference is compensated by a higher number of respondents, which could not be contacted. Overall, the percentage of successfully interviewed respondents is identical between RAT-trained and standard-trained interviewers. The fact that RAT-trained interviewers spend less time on contacting respondents and conducted fewer contact attempts with non-contacted respondents may be a consequence of the training or the non-random-allocation of designated respondents to interviewers. Three facts suggest an effect of non-random allocation: The effect of increased non-contact rates for RAT-interviewers can only be shown in urban areas. The overall time difference between first and last contact attempt to a designated respondent is smaller for RAT-interviewers than for standard interviewers (22 vs. 32 days) although the RAT-interviewers spent more time (92 vs 60 days) conducting interviews. The workload of RAT-interviewers is higher than for standard interviewers. So a non-random allocation of RAT-interviewers seems to be plausible: The RATinterviewers were already experienced interviewers and experienced and successful interviewers were especially at the end of field period more often used for the harder cases. 1 The higher workload of RAT-interviewers may be interpreted as supporting evidence. The non-random allocation may also be the explanation for the set of strange interaction effects of town size on final disposition codes. In order to clarify the effects of a Refusal Avoidance Training, a further experiment is needed. A higher number of interviewers and a random allocation would be desirable. If this is not possible, at least a matched allocation of urban areas to interviewers and similar field periods for experimental and control group are desired. Furthermore, the training should be done by a professional survey methodologist in the absence of observers. 1 This policy was conceded in a personal communication of a representative of the field agency. 9
Since the implementation of such an experimental protocol may be difficult, the research group is currently engaged in implementing a random allocation experiment based on a self-teaching multi-media-version of the RAT for CATI-interviews (Schnell/Dietz 2006). References Groves, R.M./McGonagle,K.A. (2001): A Theory Guided Interviewing Training Protocol Regarding Survey Participation; in: Journal of Official Statistics, 17, 2, p. 249-265. Schnell,R. (2004): Refusal Aversion Training (RAT) von Interviewern zur Erhöhung von Responseraten, Research Proposal to the German Research Foundation, unpublished manuscript, University of Konstanz, Center for Quantitative Methods and Survey Research. Schnell,R./Dietz,C. (2006): CATI-RAT. Multimediale Interviewer-Schulung für CATI- Interviewer, unpublished manuscript, University of Konstanz, Center for Quantitative Methods and Survey Research. Survey Research Center (2004): Interviewer Project Manual: Survey of Health, Ageing and Retirement in Europe (SHARE), Institute for Social Research, University of Michigan. Contact Comments may be sent to the authors at Rainer.Schnell@uni-konstanz.de 10