Response Scales in CATI Surveys. Interviewers Experience Wojciech Jablonski University of Lodz Poland wjablonski@uni.lodz.pl 5th ESRA Conference Ljubljana, Slovenia 17th July 2013
AGENDA Introduction Method Results Conclusions Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 2
AGENDA Introduction Method Results Conclusions Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 3
CATI CHARACTERISTICS There is no visual contact between the interviewer and the respondent its has an impact on the quality of the obtained data (de Leeuw 2002) satisficing (Krosnick 1999) - CATI interviewers are much more constrained than f2f interviewers as far as using visual aids is concerned Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 4
CATI CHARACTERISTICS SPECIAL TECHNIQUES keyword summary (Dillman 1978) unfolding (Groves 1979) split questions (Dillman 1978; Dillman et al. 2009) branching (Malhotra, Krosnick & Thomas 2009) end-point only scales (Frey 1989; de Leeuw 2002) / with midpoint (Courser & Lavrakas 2012) pre-categorised questions Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 5
RESEARCH QUESTION What are the CATI interviewers opinions and attitudes towards different response scales formats used by survey organizations? We consider the interviewers opinions of an interview to be reliable indicators of the quality of the conversation and the validity of the obtained data Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 6
AGENDA Introduction Method Results Conclusion Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 7
METHOD Data collection: Nov 2009 Aug 2010 12 major Polish survey research companies 4P research mix ASM ARC Rynek i Opinia Expert-Monitor (at present: Kantar Media) GfK Polonia IMAS International IPSOS IQS Group MillwardBrown SMG/KRC PBS DGA TNS Pentor (at present: TNS Poland) TNS OBOP (at present: TNS Poland) N of cases = 846 interviewer questionnaires & 32 IDIs Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 8
AGENDA Introduction Method Results Conclusion Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 9
RESULTS ADV/DISADV The job of a telephone interviewer, as any other job, has both disadvantages and advantages. Below there are several features of CATI interviewer s job listed. For each of them, say to what extent you consider it an advantage or a disadvantage. Use the scale of 1 to 5 shown below, where 1 means this feature is a disadvantage and 5 means that it is an advantage. The other values on the scale may be used to express opinions in-between the two extremes. Communication with the respondent is verbal only there is no eye contact 14% 35% 48% disadvantage 1---------------2---------------3---------------4---------------5 advantage 3% hard to say Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 10
RESULTS GENERAL TRAINING Below is a list of different elements, which make up the general training organised for interviewer candidates. For each of them, state to what extent, in your opinion, is it useful for participants. Use the scale of 1 to 5 shown below, where 1 means this issue is completely redundant (and so there is no point in discussing it) and 5 means it is very useful (and so discussing it is necessary). The other values on the scale may be used to express opinions in-between the two extremes. Information about the types of questions used in CATI scripts 5% 16% 78% redundant 1---------------2---------------3---------------4---------------5 necessary element element 1% hard to say Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 11
RESULTS GENERAL TRAINING ELEMENT Information about the types of questions used in CATI scripts Information about the research company (outline of the history, scope of operation) NECESSARY (points 4-5) 78% 47% Information about the ESOMAR code regulations 55% Information about the Data Protection Act 65% Information about the methods of sampling in telephone research Information about the methods of selecting respondent within a household 66% 72% Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 12
RESULTS PROBLEMS Situation considered to be particularly troublesome: handling respondents who have difficulty in understanding the questions According to the interviewers the questions used in CATI scripts are often formulated using complicated vocabulary and syntax. The questions have not been adapted to suit the intellectual skills of an average respondent Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 13
Sometimes questions are so long that when I finish reading them I don t understand what the beginning is. The respondent without written text all the more. RESULTS PROBLEMS Some questions are just too clever. Even when I look through them I have to really get into them. ( ) The questions are not designed for ordinary people but for the ones better educated. Imagine a question in which you have 39 items, an 11 point scale and you must read it 4 times for different companies. It s a nightmare. When you conduct such interviews, you can notice a moment when the respondents let their minds wander, they give answers at random. Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 14
RESULTS Is STRATEGIES A majority of interviewers claim that they sometimes deviate from interviewing rules Unfolding questions with semantic (fully-labeled) scales Transforming fully-labeled scales into end-point only scales Transforming check-all-that-apply questions into item-based yes-or-no questions Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 15
AGENDA Introduction Method Results Conclusion Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 16
CONSLUSION Interviewers demonstrate good methodological intuition they recognize questions that according to professional guidelines should and should not be used in CATI scripts The design of some questions used in CATI scripts do not follow these regulations. As a result, interactional problems between the interviewers and the respondents occur. The interviewers sometimes solve such problems by destandardizing the interview protocol Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 17
REFERENCES Courser, M.W., Lavrakas, P.J. (2012). Item Nonresponse and the 10-Point Response Scale in Telephone Surveys. Survey Practice 5(4): 1-5. de Leeuw, E.D. (2002). Data Quality in Mail, Telephone and Face to Face Surveys. Amsterdam: TT-Publikaties. Dillman, D.A. (1978). Mail and Telephone Surveys. The Total Design Method. New York: John Wiley & Sons. Dillman, D.A., Smyth, J.D., Christian, L.M. (2009). Internet, Mail and Mixed-Mode Surveys. The Tailored Design Method. Hoboken: John Wiley & Sons. Frey, J.H. (1989). Survey Research by Telephone. Newbury Park: Sage Publications. Groves, R.M. (1979). Actors and Questions in Telephone and Personal Interview Surveys. Public Opinion Quarterly 43(2): 190-205. Krosnick, J.A. (1999). Survey Research. Annual Review of Psychology 50: 537-567. Malhotra, N., Krosnick, J.A., Thomas, R.K. (2009). Optimal Design of Branching Questions to Measure Bipolar Constructs. Public Opinion Quarterly 73(2): 304-324. Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 18
PUBLICATIONS CONCERNING THE PROJECT Jablonski, W. (2012). Stressful Situations in Telephone Interviews. Survey Practice 5(4): 1-6 http://www.surveypractice.org/index.php/surveypractice/article/view/32/pdf Jablonski, W. (2012). CATI Interviewers Job Satisfaction Level. Proceedings of the Survey Research Section of the American Statistical Association: 5739-5746 http://www.amstat.org/sections/srms/proceedings/y2012/files/400237_500704.pdf Jablonski, W. (2012). Landline versus Cell Phone Surveys: Interviewers Experience. Proceedings of the ISA RC338th International Conference on Social Science Methodology: http://conference.acspri.org.au/index.php/rc33/2012/paper/view/555/17 Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 19
THANK YOU! Wojciech Jablonski wjablonski@uni.lodz.pl Wojciech Jablonski Univ. of Lodz, PL Response Scales in CATI Surveys. Is Experience 20