Mode of delivery, but not questionnaire length, affected response in an epidemiological study of eating-disordered behavior

Transcription

1 Journal of Clinical Epidemiology 57 (2004) Mode of delivery, but not questionnaire length, affected response in an epidemiological study of eating-disordered behavior J.M. Mond a, *, B. odgers b, P.J. Hay c, C. Owen d, P.J.V. Beumont e a Neuropsychiatric esearch Institute, 700 First Ave South, Fargo, North Dakota 58103, USA b Centre for Mental Health esearch, The Australian National University, Canberra ACT 0200, Australia c Discipline of Psychiatry, School of Medicine, James Cook University, Townsville QLD 4811, Australia d Academic Unit of Psychological Medicine, Medical School, Australian National University, Canberra ACT 2606, Australia e Department of Psychological Medicine, University of Sydney, Sydney NSW 2006, Australia Accepted 25 February 2004 Abstract Background and Objectives: The effects of questionnaire length and mode of delivery on response rates were examined in an epidemiological study of eating-disordered behavior. Methods: Short (8 pages) and long (14 pages) questionnaires were posted or hand-delivered to a community sample of 802 women. Nonrespondents who received the first questionnaire by hand delivery received a reminder letter and replacement questionnaire by post; those who received the initial questionnaire by post were further randomized to receive the first reminder by hand delivery or by post, in short or long form. A second reminder letter and questionnaire (in short or long form) were posted to all remaining nonrespondents. esults: The overall response rate was 52.9%. This is a conservative estimate of true response, because in a substantial proportion of cases (12.2%) individuals were no longer resident at the listed address. There was a significant effect of mode of delivery on response, favoring hand delivery, at both the initial mailout and first reminder. There was no effect of questionnaire length on response to the initial mailout, although overall response was significantly higher for the longer form. It was estimated that an overall response of 58.0% would have been achieved had first reminders been hand-delivered to all nonrespondents who received the initial mailout by post. Conclusions: Delivery of questionnaires by hand may be an effective way to increase response rates in epidemiological research, but little is to be gained by reducing questionnaire length Elsevier Inc. All rights reserved. Keywords: Postal surveys; Questionnaire length; Personal contact; esponse rates 1. Introduction In epidemiologic and public health research, the necessity for large sample sizes typically dictates the collection of information by means of mailed self-report questionnaires, because this is the most cost-effective method [1]. Mail surveys are often associated with poor response rates, however, increasing the likelihood of nonresponse bias [2]. For this reason, researchers have examined the effectiveness of a range of strategies designed to increase response rates to postal questionnaires [3]. Findings from several published reviews of the literature indicated that factors most consistently associated with higher response rates were: the use of financial incentives; use of reminder letters or telephone follow-ups (or both); use of certified mail; questionnaire content likely to be of interest * Corresponding author. Tel.: ; fax: address: jmond@nrifargo.com (J. Mond). Deceased. to participants; prenotification; and university sponsorship questionnaire content of a sensitive nature was likely to reduce response [3 8]. Some of the factors associated with superior response also entail greater cost. For example, use of telephone follow-up or certified mail is more expensive than standard mail followup, other factors being equal, and research designs incorporating face-to-face interviews are even more expensive [1]. In these cases, a trade-off exists between the benefits of higher response rates and the increased costs incurred in achieving them. One method that has not been used previously involves delivery of questionnaires to a participant s home by project staff. The fact that response is improved through the use of certified mail suggests that such a strategy may be worth investigating. Evidence concerning the effect of questionnaire length on response rates is inconsistent. Findings from some early studies suggested an optimal length of 12 pages for surveys of the general public, after which response decreased with increases in length [1]. Other findings suggested an effect /04/$ see front matter 2004 Elsevier Inc. All rights reserved. doi: /j.jclinepi

2 1168 J.M. Mond et al. / Journal of Clinical Epidemiology 57 (2004) of questionnaire length (i.e., reduced response) for surveys of more than 4 pages [5,7]. Findings from more recent studies, however, suggest little or no effect of questionnaire length [8 12]. Hoffman et al. [10] found that response rates for a 4-page questionnaire were no different from those obtained with a 16-page questionnaire, and Kalantar and Talley [11] found that response to a one-page survey was not significantly higher than that for a 7-page survey. These findings have implications for the use of a two-phase epidemiological design, as in studies of low-prevalence psychiatric disorders, in that the duration of the more costly interview phase may be minimized through collection of additional information at the screening phase [13]. We are not aware of any research to examine factors affecting response rates in general population surveys of eating disorders. In conducting pilot work for a two-phase epidemiological study of eating-disordered behavior, we examined the effect of questionnaire length and mode of delivery on response rates at the first phase. It was predicted that response rates would be higher for questionnaires delivered to the participant s place of residence by hand than for questionnaires delivered by standard mail. Consistent with the findings of recent studies, it was also expected that there would be no effect of questionnaire length on response. There was no basis on which to predict an interaction between mode of delivery and questionnaire length. 2. Materials and methods 2.1. Design and participants The study, referred to as the Health and Well-Being of Female ACT esidents project, was conducted in the Australian Capital Territory (ACT) region of Australia (population 314,000), a highly urbanized area that includes the city of Canberra. The study sample comprised 802 female ACT residents aged 18 45, selected at random from the Electoral oll and stratified by age in 5-year bands. Voting is compulsory for Australian citizens aged 18 years and above, so the Electoral oll provides a convenient singlestage sampling frame; however, incorrectly listed addresses may occur in up to 15% of cases [14]. For the initial contact ( initial mailout ), a 2 2 factorial design was used to manipulate the two independent variables, questionnaire length and mode of delivery. The short version of the questionnaire (8 pages) included a screening instrument for eating disorders, sociodemographic information, and measures of general psychological distress, disability, and quality of life [15]. The longer version of the questionnaire (14 pages) included each of these measures, as well as measures of exercise behavior, personality, and health service utilization. Of the 802 total participants, 401 received the short version of the questionnaire and 401 received the long version. Within each of these groups, half of the participants received the questionnaire by post (postal delivery, n 200); questionnaires were hand-delivered to the remaining participants (hand delivery, n 201). Participants were allocated to groups using the SPSS andom Sample procedure (SPSS, version 10.0, Chicago, IL, USA). A first reminder letter was sent to nonrespondents 8 weeks after the initial mailout. eplacement questionnaires were posted to those individuals who received the initial questionnaire by hand delivery. Among those nonrespondents who received the initial questionnaire by post, half received a replacement questionnaire by post and the remaining half received a second copy of the questionnaire by hand delivery. emaining nonrespondents received a second reminder letter (and replacement questionnaire) by post after a further 8 weeks. Each person received only one version of the questionnaire (i.e., short or long) throughout the study. The midpoint of the 2-week interval over which hand delivery of questionnaires and reminder letters was completed was used as the reference point for dispatch of postal questionnaires and for the initiation of reminder letters. The study design is outlined in Fig. 1. For questionnaires delivered by hand, up to six call-backs were used to make contact with the participant or another resident at the listed address. Deliverers were instructed to briefly explain the purpose of the study, to emphasize the importance of a good response, and to thank the person for their time. Where a member of the participant s family, but not the participant, was available to receive the questionnaire, a similar speech was made, with the request that the information be conveyed, with the questionnaire, to the participant. No attempt was made to initiate completion of the questionnaire at the time of the visit. All questionnaires were enclosed in A4 envelopes with a cover letter, instruction sheet, and a reply-paid, self-addressed return envelope. The stationery reflected sponsorship of the study by the local hospital and affiliated university department. All deliverers were female. Where an individual selected for participation was no longer resident at or was indefinitely away from the listed address, or where mailed questionnaires were returned not at this address, the selected participant was deleted from the data file without replacement. esponse was calculated as the number of completed questionnaires returned divided by the number of questionnaires posted or delivered, at each stage. Incorrectly listed addresses were not taken into account in the denominator [8] because these were less likely to be identified among participants with whom no direct contact was made, as can be seen in Fig Statistical analysis Logistic regression was used to examine the effects of questionnaire length and mode of delivery on the odds of receiving a completed questionnaire at each stage. For the initial mailout, it was expected that a response rate of approximately 30% would be achieved for questionnaires delivered by standard post; a response rate on the order of 40% was expected for questionnaires delivered by hand. A

3 J.M. Mond et al. / Journal of Clinical Epidemiology 57 (2004) Total sample (n= 802) Initial Mailout short version (n= 201) long version (n= 201) short version (n= 200) long version (n= 200) epl. (n= 67) Inc. (n= 29) efus. (n= 3) epl. (n= 76) Inc. (n= 20) efus. (n= 2) epl. (n= 58) epl. (n= 58) Inc. (n= 6) Inc. (n= 5) efus. (n= 2) emaining (n= 102) emaining (n= 103) emaining (n= 136) emaining (n= 135) 1 st eminder short version (n= 102) long version (n= 103) short version (n= 68) short version (n= 68) long version (n= 68) long version (n= 67) epl. (n= 21) Inc. (n= 0) efus. (n= 2) epl. (n= 29) epl. (n= 13) epl. (n= 18) Inc. (n= 0) Inc. (n= 3) Inc. (n= 16) efus. (n= 2) efus. (n= 2) efus. (n= 2) epl. (n= 13) epl. (n= 30) Inc. (n= 10) efus. (n= 2) 2 nd eminder short version (n= 79) long version (n= 72) short version (n= 50) short version (n= 32) long version (n= 54) long version (n= 25) epl. (n= 6) Inc. (n= 2) Inc. (n= 0) Inc. (n= 0) epl. (n= 7) epl. (n= 7) epl. (n= 4) Inc. (n= 0) epl. (n= 11) epl. (n= 6) No response (n= 71) No response (n= 65) No response (n= 43) No response (n= 28) No response (n= 42) No response (n= 18) Fig. 1. Study design. (, randomisation; epl., replied; Inc., incorrect address; efus., refused participation.) difference of this magnitude would be detected at the.05 level with power of 83% (two-sided). Analysis was conducted using SPSS version esults Cumulative response rates are shown in Table 1. Completed questionnaires were received, following reminder letters, from 424 respondents, an overall response of 52.9%. Table 1 Cumulative response rates as a function of questionnaire length and mode of delivery esponse rate, % Initial mailout First reminder Second reminder Long form Hand delivery Postal delivery Short form Hand delivery Postal delivery Main effects Hand delivery Postal delivery Long Short Total Mode of delivery refers to the use of hand delivery or standard post at the initial mailout. Cumulative response for postal delivery reflects the influence of hand-delivered first reminders among a subgroup of participants who received the initial questionnaire by post (see Fig. 1). This is a conservative estimate of true response because data from participants receiving questionnaires by hand delivery at the initial mailout (n 402) indicated that 12.2% of participants were no longer resident at the listed address (Fig. 1). Individuals aged years were overrepresented among participants with incorrectly listed addresses (n 94), relative to the remainder of the sample (28.9% vs. 12.7%), and individuals aged were underrepresented among those with incorrectly listed addresses (3.2% vs. 18.8%) (χ , df 5, P.001). Partial nonresponse was negligible ( 1%) under all conditions. esults of the logistic regression analyses are summarized in Table 2. esponse rates to the initial mailout were higher for questionnaires delivered by hand (35.6% vs. 29.0%; P.05); there was no effect of questionnaire length (long: 33.4%; short: 31.2%). The interaction between questionnaire length and mode of delivery was also nonsignificant. There was a highly significant effect of mode of delivery following the first reminder among individuals who did not respond to an initial postal questionnaire, favoring hand delivery over postal delivery (35.6% vs. 19.1%; P.004) (Fig. 1). The effect of mode of delivery was more pronounced for the long form than for the short form of the questionnaire (long: 44.8% vs. 19.1%; short: 26.5% vs. 19.1%); however, this trend did not reach statistical significance (P.16). As a result of the effect of mode of delivery at the first reminder, the initial effect of mode of delivery was no longer significant at this point (48.0% vs. 47.5%). Overall response was significantly higher for the longer form following the first

4 1170 J.M. Mond et al. / Journal of Clinical Epidemiology 57 (2004) Table 2 esults of the logistic regression analyses examining the effect of questionnaire length and mode of delivery on response rates at different stages of the survey Stage Odds ratio (95% CI) Significance 1. Initial mailout Questionnaire length 1.10 ( ).52 Mode of delivery 1.35 ( ).05 Length Mode 1.22 ( ) First reminder a Questionnaire length 1.34 ( ).04 Mode of delivery 1.02 ( ).89 Length Mode 1.10 ( ) First reminder b Questionnaire length 1.50 ( ).155 Mode of delivery 2.29 ( ).004 Length Mode 2.25 ( ) Second reminder Questionnaire length 1.44 ( ).01 Mode of delivery 0.88 ( ).35 Length Mode 1.00 ( ).99 Mode of delivery refers to the use of hand delivery or standard post at the initial mailout, except for the third analysis, where this refers to mode of delivery at the first reminder (see Fig. 1). Questionnaire length: short 0, long 1; mode of delivery: posted 0, hand-delivered 1; response status: nonresponse 0, response 1. a Includes data for all participants who received a first reminder (n 476). b Includes data for participants who received a first reminder and who received the initial questionnaire by post (n 271). reminder (51.4% vs. 44.1%; P.04) and remained so following the second reminder (57.4% vs. 48.4%; P.01) (Table 2). 4. Discussion There were two main findings. First, response rates were higher for hand-delivered questionnaires than for questionnaires delivered by standard post. This effect reached significance at the initial mailout, and was particularly noticeable at the first reminder among participants who did not respond to an initial postal questionnaire. Second, there was no effect of questionnaire length on response rates at the initial mailout, and overall response was significantly higher for the longer form. Because first reminder letters were hand-delivered to only half of those individuals who received the initial questionnaire by post, it was not possible to directly compare the effect on response of making contact with participants at the initial mailout and at the first reminder. Extrapolating from the data, however, it was estimated that an overall response of 58.0% (not taking incorrect addresses into account) would have been achieved had first reminders been hand-delivered to all nonrespondents who received the initial mailout by post and that, in the absence of personal contact with participants, at least one additional (i.e., third) mailout would have been required to achieve this outcome. Use of hand delivery at the initial mailout yielded an overall response of 51.2%. Therefore, initial use of standard post, followed by hand delivery of first reminder letters, may be the optimal method in terms of response rates. This method is also more cost efficient than the use of hand delivery at the initial mailout, the cost per questionnaire for hand delivery being approximately four times that of standard post (A$1.80 vs. 0.45). The use of certified mail has consistently been shown to yield superior response to that of other methods of postage, even more costly methods [3 6,16 18]. There is also some evidence that response rates are higher when forms are completed in the participant s home [19]. Typically, the use of (the more costly) certified mail is confined to the third and final stage of follow-up, at which point response would otherwise be poor [5,20]. Similarly, it is unlikely that hand delivery would be cost-effective in large-scale epidemiological studies if used at the initial mailout. Further, a large number of delivery staff would be required to ensure that delivery of questionnaires was effected in a reasonable period of time. Hand delivery may, however, be a viable alternative to the use of certified mail at a later stage, particularly if the number of approaches is limited by the conditions of ethics committee approval. The advantage of involving project staff in the delivery process is that this provides the opportunity to explain the study aims and to convey the importance of a good response in person. Comparison of the cost effectiveness of these different methods at different stages of follow-up would be a useful contribution to future research. It is often taken for granted in epidemiological research that efforts to reduce questionnaire length will be rewarded with improved response, but there was no evidence for this assumption in the findings of the present study. There was no effect of questionnaire length on response to the initial mailout; overall response was significantly higher for the longer form, due to a superior response at the first reminder (30.3% vs. 21.8%) and, to a lesser extent, at the second reminder (15.9% vs. 10.6%). Although it is possible that a negative effect of increased length would have been observed had the short version of the questionnaire been even shorter, several recent studies have found no effect of questionnaire length, even where a marked difference exists in the lengths of the respective versions [10 12]. These findings, together with those of the present study, suggest that efforts to improve response rate by reducing questionnaire length may be of limited benefit and that the role of other factors is more worthy of attention [9]. The finding that overall response was higher for the longer form should be interpreted with caution, because this effect appeared to be confined to questionnaires delivered by hand. We are aware of only one other study in which an effect of questionnaire length on response, favoring the longer questionnaire, has been reported. Champion and Sear [21] found that response rates were significantly higher for a 9- page questionnaire than for either 3-page or 6-page forms.

5 J.M. Mond et al. / Journal of Clinical Epidemiology 57 (2004) Interestingly, an attempt was made in their study to control for an effect of questionnaire content by using the same set of questions, presented in more or less expanded form, in each questionnaire. One interpretation of this finding is that participants attached greater importance to what was perceived as the more comprehensive survey. Similarly, in the present study, it is possible that hand delivery had a relatively greater impact for the longer form due to a perception that this mode of delivery was inappropriate for a brief selfreport questionnaire. As in other Australian studies using the Electoral oll [14], a substantial proportion of individuals were no longer resident at the listed address. Therefore, the overall response of 52.9% is a conservative estimate of true response. What constitutes an acceptable response rate in psychiatric epidemiology research is a matter of opinion [22], though lower response rates are not necessarily prone to nonresponse bias [15]. Of note is the finding that individuals aged were overrepresented, and those aged underrepresented, among participants with incorrectly listed addresses. Such differences may need to be taken into account when the Electoral oll is used and if it is important that the age distribution of respondents be tightly controlled. Aspects of the study design may limit generalization to other research. In particular, the 8 weeks between initial mailout and first reminder, and between first and second reminders, was longer than typically used. It is not clear whether response rates would have been different under a more conventional time frame. Also, participants were not given prior notice that questionnaires might be delivered to their place of residence. In the great majority of cases, use of this method was well received; however, a small number of participants refused to accept delivery of the questionnaire and one individual registered a complaint with the Ethics Committee. Overall response might have been higher had prior notice been given [7,3]. In conclusion, delivery of questionnaires by hand may be an effective way to increase response rates in epidemiological research. The additional cost involved in implementation of this strategy, and the decision as to whether this cost is justified, will depend on the particular aims and circumstances of the research and the stage of the survey at which the method is used. It appears that little is to be gained by reducing questionnaire length at the first phase in two-phase epidemiological studies of eating disorders. Acknowledgments The research was conducted while the first author was in receipt of a esearch Training Fellowship from the New South Wales Institute of Psychiatry. Thanks also to Ailsa Korten, who assisted with the statistical analysis, and to Tanya Caldwell and Jo Medway, who assisted with the coordination of hand-delivered questionnaires. eferences [1] Dillman DA. Mail and telephone surveys: the total design method. New York: Wiley; [2] Sheikh K, Mattingly S. Investigating non-response bias in mail surveys. J Epidemiol Community Health 1981;35: [3] Edwards P, oberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz, Kwan I. Increasing response rates to postal questionnaires: systematic review. BMJ 2002;324: [4] Linsky AS. Stimulating responses to mailed questionnaires: a review. Public Opin Q 1975;39: [5] Heberlein TA, Baumgartner. Factors affecting response rates to mailed questionnaires: a quantitative analysis of the published literature. Am Sociol ev 1978;43: [6] Goyder JC. Further evidence on factors affecting response rates to mailed questionnaires. Am Sociol ev 1982;47: [7] Yammarino FJ, Skinner SJ, Childers TL. Understanding mail survey response behavior: a meta-analysis. Public Opin Q 1991;55: [8] Asch DA, Jedrziewski MK, Christakis NA. esponse rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50: [9] Eaker S, Bergstrom, Bergstrom A, Adami H, Nyren O. esponse rate to mailed epidemiologic questionnaires: a population-based randomized trial of variations in design and mailing routines. Am J Epidemiol 1998;147: [10] Hoffman SC, Burke AE, Helzlsouer KJ, Comstock GW. Controlled trial of the effect of length, incentives, and follow-up techniques on response to a mailed questionnaire. Am J Epidemiol 1998;148: [11] Kalantar JS, Talley NJ. The effects of lottery incentive and length of questionnaire on health survey response rates: a randomized study. J Clin Epidemiol 1999;52: [12] Subar AF, Ziegler G, Thompson FE, Johnson C, Weissfeld JL, eding D, Kavounis KH, Hayes B; Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial Investigators. Is shorter always better? elative importance of questionnaire length and cognitive ease on response rates and data quality for two dietary questionnaires. Am J Epidemiol 2001;153: [13] Newman SC, Shrout PE, Bland C. The efficiency of two-phase designs in prevalence surveys of mental disorders. [Erratum in: Psychol Med 1990 Aug;20(3):following 745.] Psychol Med 1990;20: [14] Smith W, Mitchell P, Attebo K, Leeder S. Selection bias from sampling frames: telephone directory and electoral roll compared with doorto-door population census: results from the Blue Mountains Eye Study. Aust N Z J Public Health 1997;21: [15] Mond JM, odgers B, Hay PJ, Owen C, Beumont PJV. Nonresponse bias in a general population survey of eating-disordered behavior. Int J Eat Disord 2004;36: [16] imm EB, Stampfer MJ, Colditz GA, Giovannucci E, Willett WC. Effectiveness of various mailing strategies among nonrespondents in a prospective cohort study. Am J Epidemiol 1990;131: [17] Del Valle ML, Morgenstern H, ogstad TL, Albright C, Vickrey BG. A randomized trial of the impact of certified mail on response rate to a physician survey, and cost-effectiveness analysis. Eval Health Prof 1997;20: [18] Gibson PJ, Koepsell TD, Diehr P, Hale C. Increasing response rates for mailed surveys of Medicaid clients and other low-income populations. Am J Epidemiol 1999;149: [19] Picavet HSJ. National health surveys by mail or home interview: effects on response. J Epidemiol Community Health 2001;55: [20] Dillman DA. The design and administration of mail surveys. Annu ev Sociol 1991;17: [21] Champion DJ, Sear AM. Questionnaire response rates: a methodological analysis. Soc Forces 1969;47: [22] Williams P, Macdonald A. The effect of non-response bias on the results of two-stage screening surveys of psychiatric disorder. Soc Psychiatry 1986;21:182 6.