Effects of Data Collection Methods On Participant Responses In Television Research Studies

Transcription

1 Cairo University Faculty of Mass Communication Radio and Television Department Effects of Data Collection Methods On Participant Responses In Television Research Studies A Comparative Study A Summary of TheThesis Submitted in Fulfillment of the Requirements For Degree of Masters in Mass Communication Summary By/ Amira Abdel Moniem Abdel Fattah Abdel Samad Under the supervision of/ Dr. Amany El SayedFahmy January 2016

2 ABSTRACT Askingquestionsoffrespondentsisoneofthemaindatacollectionmethodsinsocialscience anditsassociatedappliedfields.theoldestsurveymethodsaretheface-to-faceinterviewand th questionnaire.after1970,telephoneinterviewshavebecomeincreasinglypopular.oneofthem ostimportantquestionsforsurveyresearchersiswhetherthedatacollectedbyonemethoddifferfromthe datacollectedby another.thisstudycomparesfourmajormodesofsurveyresearch:face-tofaceinterviews, telephoneinterviews,self- administered paper and pencil questionnaires andonlinequestionnaires.afteratheoreticaldiscussionwhymodeeffects mayoccur,theresearcherpresentsacomprehensiveoverviewofthe researchliterature.thisisfollowedbytheresultsofa mode comparison.thecombinationofanincisiveanalysisofissuesinsurveymethodologywithsophisticated dataanalysistechniquesgivesthisstudyabroadscope.itwillbeofinteresttobothsocial sciencemethodologistsandpeoplewhoworkintheoreticalorapplied socialresearch. Key words:data collection methods, data quality; mixed mode; survey error; mode effects; satisficing; social desirability bias.

3 Introduction The research on the effects of survey mode, and Internet in particular, on survey responses and data quality is attracting a great deal of attention in the survey methodology literature (e.g., Cooper, 2008; Couper and Miller, 2008; Baker et al., 2010). During the history of survey research, the field has witnessed many transitions in the uses of various modes of data collection. Survey methodologists make the point that data is a product of the collection process, i.e., generated at the time of the interview or completion of the questionnaire, rather than just being there to be collected (implying that data collection is a misleading term) (Groves et al., 2004). In addition to the so-called measurement effects, arising from the process of responding to questions in different survey modes. Traditionally in surveys data were collected either by an interviewer visiting a respondent known as the face-to-face (FTF) interviewing yielding high response rates, permitting the development of good rapport between interviewers and respondents, and allowing the use of visual aids that facilitated the measurement process. Face-to-face interviews have been the recommended gold standard for surveys in general and social studies research in particular (Mitchell and Carson, 1989; Arrow et al., 1993). The face-to-face interview is one of the oldest forms of data collection in surveys, and it has evolved from a short and simple inquiry in the thirties into a complex and highly flexible research instrument (Rossi, Wright &Anderson, 1983; Smith, 1987). Because of its flexibility and great potential, the face-to-face interview has long been considered a superior data collection technique, or through a self-administered mail questionnaire the data collected by mail surveys have often been considered suspect unless proven otherwise. This is exactly the opposite of the view held toward the accepted face-to-face interview (Dillman, 1978, p.1).

4 In the second half of the twentieth century this picture changed rapidly. Mail and to some extent telephone surveys have been much more used in practice; mostly for reasons of lower cost. Telephone surveys became increasingly popular; reaching their peak in the 1990s. This is caused by improved technology, by the development of random digit dialing as a sampling technique, but, above all by the increased availability of and access to telephones for the general public. Nevertheless, although the telephone interview has attained an increasing significance in the daily practice of data collection, it also had to prove itself against the generally accepted face to face interview (Körmendi&Noordhoek, 1989; Sykes &Collins, 1988). This has led to a renewed interest in alternatives for face-to-face interviews, and a renewed research effort to optimize mail and telephone surveys. Advances in computer technology in the last thirty years have made computer-assisted survey methods possible, including methods for Internet surveys. With the number of internet users having increased very rapidly since its introduction, the applicability and significance of this new technology has become of great importance to many public opinion researchers (Couper 2000). The current trend in social studies research, like in other survey based research, however, is to collect data using the Internet (Thurston, 2006). Sophisticated questionnaires can be delivered to large samples on record time at fairly low costs. Judging from the current growth in Internet penetration rates and use, Internet also has the potential to overcome the primary concern about population coverage and representativeness to become the mode of choice for survey data collection in the not so distant future (see, e.g., Couper (2005). This variety of data collection methods led to methodological questions, such as, which method to choose? Which is best?

5 The broad areas of inquiry were attempted on the mode comparison between face-to-face interviews, telephone questionnaires and self-administered paper and pencil questionnaires (SAQ) and online questionnaires: response pattern (response rate, and completion rate); and data quality (item-non response, response time, number of complete statements of open-ended questions, levels of satisficing and social desirability bias). Given that the interest in online surveys is relatively recent phenomenon and that there is much functional similarity between the four data collection methods, the researcher incorporated the literature on online surveys in conducting this study. Research Problem and Significance The choice of mode has natural consequences, the problem is not whether differences between modes exist but how great are the differences, and why do they exist, and what effect will they have on the uses that are made of the data? Similarly, Biemer pointed out that statistical significance of a comparison alone is not necessarily indicative of a data quality differential between two surveys (1988, p.276). Researchers are constantly seeking the most effective means of collecting data. The new technological age has provided researchers with many options to collect their data, but are the respondents willing to adopt these alternative modes? The use of surveys is a vital tool for use by social science researchers. Surveys are used to understand perceptions, collect information, and gauge opinions. With the continuous increase in the extensive time that it takes to transfer data from a paper survey to an electronic database, surveyors are beginning to turn to the World Wide Web to save both time and money. In the academic arena, there has been very little research to verify that the demands of high response

6 rates to control for non-response error are being met. Also, research has been limited in determining whether there is a significant difference in data quality in answers of electronic surveys versus traditional surveys. This study is designed to determine which data collection mode is a more reliable means of collecting data from the given population on a sensitive topic and if there is a difference in response rates based on the mode of survey delivery. Social science researchers have noted that the medium in which a researcher gathers data may affect the data gathered (Babbie, 1998). This effect may skew the type of the information gathered or whether the audience self-selects in a particular way. A number of studies have examined the benefits and liabilities of various data collection methods including personal interviews, telephone interviews, and Self-Administered Questionnaire however, few studies have examined but how great the differences are and why do they exist, and what effect they will have on the data quality obtained and the uses of this data in any scientific research. Another aspect is the impact of data collection mode, on the response rate to the data collection method. It is important to conduct such research as the information regarding the respondents willingness to respond to different modes of data-collecting techniques should be helpful in creating improvements in survey methodology for researchers in the media field. Aims and Objectives Individual researchers are now faced with a difficult decision when selecting a data collection method for their survey. Besides, other factors enter into this complex decision process such as the population under study,and the questionnaire content. One of the aims of this current study is to carry out a literature review to look at mode effects. Another aimof this study is to examine the effects of data collection methods on the quality of the resulting data and on substantive

7 conclusions based on those data, and to identify causesof observed mode differences, in particular to disentangle effects arising from differences in the question stimuli (i.e. how questions are asked in different modes) and effects arising from the physical presence of the interviewer and resulting aspects such as the pace with which the interview is conducted (and other characteristics of the mode not specifically controlled for, finally to examine the effects of mode of data collection on research results, especially on the consequences for the relationships between variables and to see whether interviewer presence, and question sensitivity in four different data collection methods would have an effect on quality of survey responses. The objective is to address particular problems associated with administering surveys through different modes. It involved a direct comparison among four methods employed. Research Hypothesis To assess the nature of mode effects, the researcher test specific hypotheses about how the differences between modes may lead to differences in responses. Self-administered and interviewer administered modes mainly differ in two respects: the sensory channel available for the transmission of information, in particular the physical presence of the interviewer. (See for example de Leeuw (2005), Holbrook, Green and Krosnick (2003), Fricker, Galesic, Tourangeau and Yan (2005), and Tourangeau and Smith (1996).) There is a number of potential explanations of how these differences between the four modes may affect task difficulty, respondent motivation and willingness to self-disclose and hence the response process and resulting responses. (See Krosnick 1991 for an extensive discussion of sources of task difficulty and respondent motivation, some of which may not be affected by mode.) Each of these potential explanations has empirical implications which the researcher tests in the following. The

8 hypotheses derived are mainly based on discussions by Krosnick, Narayan, and Smith (1996), Tourangeau, Rips, and Rasinski (2000), Holbrook, Green, and Krosnick (2003) and de Leeuw (1992; 2005). A number of hypotheses were developed based on past research findings concerning response pattern and response quality and response bias of the data collection methods. Those hypotheses are shown in this section, grouped under the various areas of interest. The researcher formulated the following research hypothesis: Response Pattern Response Rate Academic scholars such as Baruch (1999) assert that response rates for academic studies have demonstrated a general decline in recent years. Unfortunately, researchers found that survey mode does impact respondent fatigue with survey tasks (Griffis, Goldsby and Cooper 2003). Moreover, some studies of online surveys show increased response rates from the ever growing online contacts (Mehda and Sivadas 1995; Smith 1997). The following hypothesis is thus given: H1: There will be a significant difference in response rates of online data collection manifesting higher response rates than face-to-face, telephone, and paper and pencil data collection methods.

9 Completion Rate Vazzana and Bachman (1994) provide empirical results that completion rates from in-person data collection methods do not differ significantly from completion rates of answers from less personal methods. Thus, the following hypothesis is given: H2: There will be no significant difference in completionrates between interviewer administered questionnaires (face-to-face telephone) and self-administered questionnaires (paper and pencil- online). Data Quality Item non-response Tse (1995) compared unanswered questions between mail and methods from his research with University of Hong Kong employees. Tse (1995) found no significant differences between the average unanswered questions between the two methods. Results from Sproull s (1986) study involving business professionals, revealed that online data collection yielded higher levels of item non-response compared to more conventional methods of data collection. Based on these findings, interviews should produce the lowest levels of non-response and online forms should produce the highest levels of non-response. Therefore, the following hypothesis is given: H3: There is a significant relation between Interviewer administered data collection methods manifesting fewer items non- response than self-administered data collection methods.

10 Response Time Leaner communication media, such as text-based forms, result in higher decision times compared to richer media forms (Dennis and Kinney 1998). White s (2003) research revealed that some respondents spent over an hour to complete survey tasks using an online-based data collection method. Furthermore, many scholars (e.g., Fowler and Wackerbarth 1980) claim that it takes longer to manually produce a message than to speak it (Dennis and Kinney 1998). Therefore, use of text-based data collection methods that require typing or writing, such as online forms and paper and pencil questionnaires, will produce higher response times compared to faceto-face data collection methods (Dennis and Kinney 1998). Hence the following hypothesis is given: H4: There is a significant relation between Interviewer administered data collection methods manifesting shorter response pace and self-administered data collection methods manifesting longer response pace. Complete statements to open-ended question While Deutskens, de Jong, de Ruyter and Wetzels (2006) survey among US and UK business professionals reveals consistent levels of reliability between online and postal mail administered surveys, these researchers found other differences between both survey methods. Specifically, online respondents provided lengthier responses to open-ended questions, providing higher levels of completeness of answer. Therefore, the following hypothesis is given:

11 H5: There is a significant relation between online surveys having longer responses to openended questions than paper and pencil questionnaires, and that, both perform better than interviewer administered face-to-face and telephone interviews. Response Bias Respondent Satisfaction Respondents are more likely to satisfice with interviewer administered modes. This hypothesis is supported by evidence reported by Holbrook, Green, and Krosnick (2003), who found more acquiescence, and no opinion responses with interviewer administered interviewing. Jordan, Marcus, and Reeder (1980) similarly reported more acquiescence, don t know and no answer responses in attitude items with telephone interviewing. No supporting evidence was found by De Leeuw (1992), who did not find differences in acquiescence, item non-response or the length of open-ended answers (possibly because the open questions were short and well defined). The following hypothesis was formulated: H6: Interviewer administered interviews will significantly manifest highest levels of satisfaction among all four data collection methodologies regarding the respondents attitudes towards explicit scenes in satellite movies Heerwegh (2009) found that acquiescence was not significantly higher in face-to-face than internet. Higher levels were found in a mixed mode design which included self-administration than in one which mixed interviewer-administered modes only (Beukenhorst and Wetzels 2009).

12 They questioned whether this was due to satisficing or not wanting to admit a lack of opinion to an interviewer Krosnick (2005). Hence this hypothesis was constructed. H6 a Acquiescence will be higher in the self-administered modes. There is inconsistent evidence of systematic differences between modes on provision of no opinion and don t know responses. Higher rates of these responses in telephone interviews than in face-to-face are reported by Krosnick (2005), Holbrook et al (2003), Ariel et al (2008) and Roberts (2007- again only in long interviews). However mail obtained higher rates of don t know than internet or interviewer-administered modes for Beukenhorst and Wetzels (2009). Watson (2009) reported higher levels by internet than telephone, Duffy et al (2005), Heerwegh (2009) and Heerwegh and Loosveldt (2008) report higher don t know and/or not sure responses from internet mode than face-to-face (significant differences but at very low rates). Whereas Roberts (2007) reports higher don t know levels to questions on scientific knowledge by telephone than internet perhaps due to greater time pressure. The type of question may be relevant, since the Duffy et al findings were with regard to sensitive questions, so this could be due to the absence of an interviewer. The following hypothesis was formed: H6 b There will be significantly higher rates of No opinion/ don t know responses in interviewer administered questionnaires. Social Desirability Bias Holbrook et al. (2003); Jordan et al. (1980); and Groves and Kahn (1979). Hochstim, 1967; Sudman and Bradburn, 1974; Aquilino, 1994 hypothesized that the greater social and physical

13 distance between the interviewer and respondent in telephone interviews compared with those conducted in-person, would make social desirability bias more likely to compromise data quality in the latter, compared with the former, and the findings of some studies support this. In other words Interviewer-administered modes may result in social desirability bias, where the respondent feels the need to respond in a socially acceptable manner to avoid judgment. Additionally, respondents may feel a lack of privacy when answering sensitive questions in the presence of an interviewer. According to this the following hypotheses were formed: H7 a : Respondents are significantly more willing give socially desirable answers when asked about their attitudes towards sensitive issues like gender equality, gender roles, and sexual content on satellite movies in interviewer administered modes. H7 b : Self reports on watching sexual content when others are around as being normal and watching sexual content on purpose, significantly increases in self-administered modes relative to administration of the same questions by an interviewer. Concise Summary of main results In (Table 20) Facetoface(F),Telephone(T), Paper and Pencil (P), and Onlinesurvey(O)areevaluatedonseveral criteria.foreachcriterionapredictionandtheresultofthestatisticaltestaregiveninthe firstandsecondcolumn.thesign">"indicatesahigherscoreonthecriterion(e.g.,better performance)and"<"indicatesalowerscore(e.g.,worseperformance).forexamplet>f

14 ontheindicatorreliability meanshigherreliability (i.e.,betterperformance).areferenceto theappropriatesectionofthischapterisgiveninthelastcolumn. Table (20): Concise Summary of main results Criterion Prediction Result Section Response Rate O>F>T>P F>P>T>O Completion Rate F=T=P=O F>T>P>O Item Non-Response O=P, F=T, O =P>T =F O&P>F&T P>O>T>F Response Time P> F=T F>T>P Complete Statements O>P>F>T P>O>F>T to Open-ended question

15 Note.Thisisaconcisesummaryoftheresultsofthestatisticaltests.Whenthemodesdidnot differona significancelevelof0.05thisisindicatedinthetable by"=".theequalsigndoesmeanthattherearenostatistical differencesbetweenthemodes,notthattheresultsarecompletelyidentical.foramoredeta ileddiscussionofthe results see the appropriate section in this chapter. OnlyinalimitednumberofcasesdidIdetectstatisticallysignificantdifferencesatthe.05- level.whenadifferencebetweenmodeswassignificantitalwaysindicateda(small) differencebetween one surveyconditionandtheotherthreeconditions.however,a small(notsignificant)trendcouldbenoticedinthepredicteddirection.allsurveymethods performedmoderatelywellonthereliabilityandscalingcriteria:theface-to-faceinterview survey methodshowed thebestresults,whiletheonlinesurveywastheleastsatisfactory.whenindividualresponsepatternswerein vestigated,asmallmodeeffectscouldbe distinguished. Fromapracticalpointofviewtheseresultsarereassuring:onlyverysmalleffects werefound.fromatheoreticalpointofview,theseresultsareslightlydisappointing.two importantfactorsweredistinguishedwhichcouldinfluencethedataquality: opportunitytorelatedifferentquestionstoeachother.theface-to-face timepressureand and telephone,didshowbetterresults.further researchseemsnecessary, since developmentsincomputersurveying are in a constant flux.

16 Summary: Askingquestionsoffrespondentsisoneofthemaindatacollectionmethodsinsocialscience anditsassociatedappliedfields.theoldestsurveymethodsaretheface-to-faceinterviewand th questionnaire.after1970,telephoneinterviewshavebecomeincreasinglypopular.oneofthem ostimportantquestionsforsurveyresearchersiswhetherthedatacollectedbyonemethoddifferfromthe datacollectedby another.thisstudycomparesfourmajormodesofsurveyresearch:face-tofaceinterviews, telephoneinterviews,self- administered paper and pencil questionnaires andonlinequestionnaires.afteratheoreticaldiscussionwhymodeeffects mayoccur,theresearcherpresentsacomprehensiveoverviewofthe researchliterature.thisisfollowedbytheresultsofa mode comparison.thecombinationofanincisiveanalysisofissuesinsurveymethodologywithsophisticated dataanalysistechniquesgivesthisstudyabroadscope.itwillbeofinteresttobothsocial sciencemethodologistsandpeoplewhoworkintheoreticalorapplied socialresearch. Key words:data collection methods, data quality; mixed mode; survey error; mode effects; satisficing; social desirability bias.