Increasing Response to Mailed Questionnaires by Including a Pencil/Pen

Transcription

1 American Journal of Epidemiology Copyright ª 2005 by the Johns Hopkins Bloomberg School of Public Health All rights reserved Vol. 162, No. 3 Printed in U.S.A. DOI: /aje/kwi194 PRACTICE OF EPIDEMIOLOGY Increasing Response to Mailed Questionnaires by Including a Pencil/Pen Emily White 1,2, Patricia A. Carney 3, and Ann Shattuck Kolar 1 1 Cancer Prevention Program, Fred Hutchinson Cancer Research Center, Seattle, WA. 2 Department of Epidemiology, School of Public Health and Community Medicine, University of Washington, Seattle, WA. 3 Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH. Received for publication December 16, 2004; accepted for publication March 22, Nonmonetary incentives lead to small increases in response rates to mailed questionnaires. However, inclusion of a pen or pencil, which may be a facilitating factor as well as a reward, has not been shown to improve response to health surveys in prior trials. In 2001 and 2002, the authors conducted two US trials in which a study-logo pen or pencil was randomly included in a second questionnaire mailed to nonresponders to a first mailing. In the first study, of 10,686 nonresponders to a cohort recruitment mailing, response to the second mailing was 55% with inclusion of a pen versus 40% without one (p < 0.001). In the second study, of 141 nonresponders to a pilot follow-up survey conducted 2 years after entry into a cohort, response was 43% with inclusion of a pencil versus 24% without one (p ¼ 0.02). This percentage point increase for mailing 2 translated to a 5 6 percentage point increase after the two mailings combined. In a simulated study of three mailings based on these studies, the overall response rate increased by 4 percentage points at no added cost through inclusion of a pencil in the second mailing. The additional cost of the pencil was compensated for by the reduced number of nonrespondents sent packets at the third mailing. This study supports including a study-logo pen or pencil in a second questionnaire mailing to nonrespondents as a cost-effective method of increasing response rates. data collection; epidemiologic methods; questionnaires Abbreviations: NHWH, New Hampshire Women for Health; VITAL, VITamins And Lifestyle. A high response rate to mailed, self-administered questionnaires is often important for minimizing selection bias and/or recruiting a representative population in an epidemiologic study (1). Very high response rates are needed to reduce bias in case-control studies, in surveys whose aim is to estimate population prevalence of exposures, and in cohort study follow-up surveys. For initial recruitment into a cohort, selection bias is usually not a concern (because participation generally cannot be influenced by joint effects of exposures and future outcomes), but cost-efficient recruitment is often a goal. Achieving high response rates at reasonable costs can often be challenging. Numerous randomized trials of methods of increasing response rates to mailed questionnaires have been conducted (1 3). In an excellent review and meta-analysis of 292 such trials, Edwards et al. (2) identified several factors that increase response rates. Approaches that had the strongest effects (odds ratios of , or approximately a percentage point increase for response rates of percent) were (from the greatest effect to the least): a more salient survey topic; recorded (e.g., certified) delivery; including a monetary incentive; a shorter questionnaire versus a longer questionnaire; including incentives along with the questionnaire versus upon return of the questionnaire; preliminary contact with potential participants; follow-up of nonrespondents; and including the questionnaire with the follow-up contacts (2). Correspondence to Dr. Emily White, Cancer Prevention Program, Fred Hutchinson Cancer Research Center, 1100 Fairview Avenue North, P.O. Box 19024, Seattle, WA ( ewhite@fhcrc.org). 261

2 262 White et al. Meta-analyses of trials of nonmonetary incentives (pencils, pens, and small gifts) show that they have a smaller positive effect on response rates in the range of a 5 8 percentage point increase, on average (2 4). The majority of the trials were of surveys in the social sciences or in business/marketing or surveys of health professionals, rather than population-based health surveys. To our knowledge, only three studies have tested whether including a pencil or pen with a population-based, mailed health questionnaire increases survey response, and none of them reported a significant increase (5 7). In this paper, we report on two randomized controlled trials in which a pen or pencil was randomly included or not included in a second mailing sent to nonresponders to the first mailing. In the first study, the survey was a recruitment mailing for a study on women s health. In the second, the survey was a follow-up mailing sent 2 years after men and women completed a baseline cohort study questionnaire. We determined how inclusion of a pen or pencil in the second mailing affected response rates and costs. We analyzed costs to understand whether the additional cost of the pen or pencil was compensated for by the reduced number of nonrespondents who would need to be contacted in a third round of mailings. MATERIALS AND METHODS Study 1 The New Hampshire Women for Health Study The primary purpose of study 1 was to determine whether inclusion of pens in study recruitment materials sent to nonresponders to an initial mailing would increase participation in a large cohort study. The New Hampshire Women for Health (NHWH) Study is being conducted to evaluate the impact of hormone replacement therapy on mammography performance (i.e., accuracy), breast cancer incidence, tumor prognostic characteristics, and health-related quality of life. Women were identified as eligible for recruitment using a population-based mammography registry, the New Hampshire Mammography Network, through which permission was given to contact participants for research purposes. Further details about the design and development of the mammography registry are provided elsewhere (8). Recruitment for the NHWH Study involved sending a personalized cover letter about the study, a consent form, and a 12-page questionnaire, which women were asked to complete and return with their consent forms by mail. After an initial mailing with no study incentives was sent in July 2001 to 39,050 women, 3,550 were found to be ineligible (because of death, moving out of the area, or an undeliverable first mailing); this left 35,500 eligible women. Of those women, 24,814 responded; thus, 10,686 women were eligible for the second-mailing study (pen vs. no pen). A pen with the study logo printed on it was included in the r of the initial recruitment materials to randomly selected (with 50 percent probability) nonresponders (n ¼ 5,364), while the remaining nonresponders (n ¼ 5,322) were r ed study recruitment materials with no pen. We stratified our response rates on the basis of when the surveys were returned (within 30 days or within days). All study activities were approved by Dartmouth College s Committee for the Protection of Human Subjects. Study 2 The VITamins And Lifestyle Study The primary aim of study 2 was to investigate the most effective strategies for minimizing cost and enhancing response rates to a follow-up questionnaire in the 77,700- person VITamins And Lifestyle (VITAL) Study cohort (9). Study 2 was conducted among Vanguard participants in the VITAL cohort. The Vanguard participants were 573 men and women in 13 counties of western Washington State who had completed the VITAL baseline questionnaire during a pilot phase of the VITAL Study in In June December 2002, approximately 2 years after completion of the baseline questionnaire, we conducted study 2 among the Vanguard participants to assess response to a follow-up questionnaire using three mailings. We excluded 40 Vanguard participants who had recently been contacted for a pilot study for DNA collection, as well as four who had died, six who had moved out of the study area, and 71 who were out of the target age range of years; this left 452 participants for the mailing study. We determined after mailing that 13 others were ineligible because of death, moving out of the study area, or having an undeliverable questionnaire with no new address being found; this left 439 eligible men and women in the mailing study. The first mailing contained a personalized cover letter, a questionnaire, and a postage-paid return envelope, which was followed 2 weeks later by a reminder postcard. A total of 298 subjects returned the questionnaire, leaving 141 nonrespondents for the randomized trial. The second questionnaire was sent to the nonrespondents 2 months after the initial mailing. Among those persons, we randomly selected a group (with 50 percent probability) to receive a study-logo pencil with the questionnaire packet. A third mailing was sent to nonresponders 2 months after the second mailing. The Fred Hutchinson Cancer Research Center s institutional review board approved this study and all related materials. Statistical analyses In study 1, we assessed differences in response rates between the pen and no-pen groups, using chi-squared tests. In study 2, we used Mantel-Haenszel methods for odds ratios and p values, to allow adjustment for sex, and converted the sex-adjusted odds ratio to sex-adjusted response rates for the pencil and no-pencil groups. For both studies, we computed an estimated response rate (RR) across the first two mailings as RR i ¼ RR m1 þ (1 ÿ RR m1 )RR m2i, where i refers to the pen/pencil group versus the no-pen/-pencil group, m1 refers to mailing 1, and m2i refers to mailing 2 in group i. We estimated how the observed differences in response rates between the groups would affect costs for a mailing protocol of up to three mailings per person, when the second mailing included a pencil. Estimated response rates for mailings 1 and 2 were based on the average from studies 1 and 2, and the estimated response rate to mailing 3 was based on study 2. Estimated costs were obtained from study 2 vendors and the US Postal Service in 2003.

3 Pencils/Pens and Questionnaire Response 263 TABLE 1. Characteristics of nonrespondents to a first mailing who were randomized to receive a pen or pencil in a second mailing versus those receiving no pen or pencil, New Hampshire Women for Health (NHWH) Study (New Hampshire, 2001) and VITamins And Lifestyle (VITAL) Study (Washington State, 2002) RESULTS Pen or pencil No pen or pencil Characteristic No. % No. % NHWH Study (n ¼ 5,364) (n ¼ 5,322) Age (years) < , , , , , , Missing data Education Less than high school High school graduate 1, , Associate degree/some college 1, , College graduate Some graduate school Missing data VITAL Study (n ¼ 73) (n ¼ 68) Sex Male Female Age (years) Table 1 shows the comparability of nonrespondents to the first mailing who were randomized to receive a pencil or pen in the second mailing with those who did not receive a pen or pencil. In the NHWH Study, the two randomized groups were similar in their distributions by age and education. In the VITAL Study, randomization failed to yield a similar sex distribution between groups; therefore, we statistically adjusted for sex in the results from this study. The two groups were comparable by age (information on educational level was not available for nonrespondents in this study). Tables 2 and 3 show the response rates for the first and second questionnaire mailings and the differences seen with inclusion of a study-logo pen or pencil in the second mailing to nonrespondents. In study 1, of the 35,500 women eligible for recruitment into the NHWH Study, 24,814 (70 percent) responded to the initial mailing, leaving 10,686 eligible for the pen incentive study with r ing of recruitment materials (table 2). Using a 30-day cutoff, 49 percent of nonrespondents to the first mailing who received the pen in the TABLE 2. Effect on response rates of including a pen in a mailing to nonrespondents in a study recruitment mailing, New Hampshire Women for Health Study, New Hampshire, 2001 Mailing 1 (no incentive) No. sent mailing* 35,500 No. completing questionnaire 24,814 Response rate (%) 70 Mailing 2 (pen vs. no pen) Pen No. sent mailing 5,364 No. completing questionnairey 2,960 Response rate (%) 55 No pen No. sent mailing 5,322 No. completing questionnairey 2,106 Response rate (%) 40 p for difference <0.001 Estimated response ratez for mailings 1 and 2 combined (%) Pen 87 No pen 82 * Excludes 3,550 women who were not eligible for this study because of death, moving out of the area, or an undeliverable questionnaire. y Completion of questionnaire within 60 days of the second mailing. z See Materials and Methods. second mailing responded to the questionnaire, as opposed to 31 percent of women not receiving a pen (p < 0.001; data not shown); these would be the appropriate figures for studies in which researchers planned to conduct a third mailing at 1 month. Within 60 days of the second mailing, these percentages were 55 percent for the pen group and 40 percent for the no-pen group (table 2; p < 0.001). The estimated cumulative response across the two mailings was 5 percentage points higher for persons receiving the pen (87 percent vs. 82 percent). In study 2, the VITAL Study, of a mailing to 439 eligible subjects who had completed a baseline questionnaire, 68 percent completed a year 2 questionnaire in response to a first mailing, which did not include an incentive (table 3). Among the 141 nonrespondents to the first mailing, the response rate was substantially higher in the group that received a pencil in their second questionnaire packet versus those who did not (43 percent vs. 24 percent; p ¼ 0.02, adjusted for sex). The effect of the pencil versus no pencil on response was somewhat greater among women (38 percent vs. 18 percent) than among men (49 percent vs. 32 percent); however, this difference by sex was not statistically significant (p ¼ 0.62 for effect of sex 3 pencil/no-pencil interaction on response) (data not shown). Estimated cumulative response across the two mailings was 6 percentage points higher for persons receiving the pen (82 percent vs. 76 percent).

4 264 White et al. TABLE 3. Effect on response rates of including a pencil in a mailing to nonrespondents in a pilot study of a year 2 cohort survey, VITamins And Lifestyle Study, Washington State, 2002 Mailing 1 (no incentive) No. sent mailing* 439 No. completing questionnaire 298 Response rate (%) 68 Mailing 2 (pencil vs. no pencil) Pencil No. sent mailing 73 No. completing questionnairey 32 Response rate (%) 44 Response ratez (%) 43 No pencil No. sent mailing 68 No. completing questionnairey 16 Response rate (%) 24 Response ratez (%) 24 p for differencez 0.02 Estimated response rate for mailings 1 and 2 combined (%) Pencil 82 No pencil 76 * Excludes 13 people who were not eligible for this study because of death, moving out of the area, or an undeliverable questionnaire with no new address being found. y Completion of questionnaire within 60 days of the second mailing. z Adjusted for sex. See Materials and Methods. Table 4 shows projected response rates and costs of a mailing protocol consisting of three questionnaire mailings, comparing mailings that include a pencil in the second mailing only with those that do not. Response numbers and costs are based on 10,000 study participants. Response rates for the first and second mailings are averages from studies 1 and 2 (tables 2 and 3), and the response rate for mailing 3 (26 percent) was based on study 2. Costs were obtained from study 2 vendors and were $1.90 for mailing a questionnaire packet, including a cover letter, a 16-page questionnaire, and outgoing and return envelopes, as well as outgoing postage and packet assembly. Return mailing costs included only the cost of return postage by business reply mail ($1.28). Additional costs for including a pencil in the second mailing were $0.18 for the study-logo pencil and a postal charge for nonmachine reading when the pencil was included in the packet. There was no additional charge for packet assembly using our vendor, nor was there additional outgoing postage when the pencil was included. Application of these costs and response rates to a thre ing strategy yielded an increased overall response rate (88 percent) for inclusion of the pencil in the second mailing as opposed to no pencil (84 percent), while costs were almost identical ($39,753 with the pencil vs. $39,698 without it). DISCUSSION The two randomized trials evaluated here found that including a pen or pencil in a second mailing to nonrespondents to the first mailing substantially increased response rates to that mailing. These studies had similar results, even though one was for a study recruitment mailing and one was for a follow-up mailing to persons who had completed a questionnaire 2 years earlier. The rate of response to the second mailing with the pen or pencil was increased by percentage points over the mailing without the pen or pencil, which is equivalent to a 5 6 percentage point increase after the two mailings. In a simulated study based on three mailings, including a pencil in the second mailing increased the overall estimated response rate by 4 percentage points without adding costs. This occurs because the enhanced response to the second mailing saves costs on the third mailing to nonresponders to the second mailing. This supports the inclusion of a study-logo pen or pencil in a second questionnaire mailing. Our results are in contrast to those of three earlier studies of whether including a pen or pencil affects response rates in epidemiologic or health surveys in the general (non-healthprofessional) population. Aveyard et al. (5) tested whether sending a pencil and eraser with a questionnaire to smokers selected from health records would increase the response rate. Including a pencil and eraser led to a nonsignificant decrease in the response rate (odds ratio ¼ 0.77, 95 percent confidence interval: 0.46, 1.29). In a follow-up mailing to persons who had recently completed baseline questionnaires and had blood drawn for the CLUE II Study, Hoffman et al. (6) found that survey response was almost identical for persons sent a printed pencil with a mailed questionnaire (47 percent) and persons sent no incentive (45 percent). Etter et al. (7) conducted a study with a three-way factorial design in which students, faculty, and staff of a Swiss university were sent a survey on smoking habits and were randomized to receive 1) a request to provide a saliva sample, 2) a ballpoint pen, and/or 3) an offer to participate in a lottery for a weekend trip. Overall, 59 percent of those randomized to receive the pen returned the questionnaire within 28 days, as opposed to 56 percent of those not sent a pen (p ¼ 0.10). In related studies, Clark et al. (10) found that inclusion of a pen did not increase response to a questionnaire sent to gynecologists, and Willimack et al. (11) reported a nonsignificant 5 percentage point increase (81 percent vs. 76 percent; p ¼ 0.29) when a gift-type ballpoint pen was sent with the recruitment letter for a face-to-face interview on social issues. It is not clear why we observed a large effect of including a pen or pencil in our two trials while these studies did not. If, as we found, inclusion of a pen or pencil does increase response, it may be because it is viewed as an incentive or a reward. Numerous studies have been conducted on nonmonetary incentives designed to increase response rates. Two meta-analyses done in the early 1990s, each of which reviewed 12 (overlapping) studies, found that nonmonetary incentives improved response rates by percentage

5 Pencils/Pens and Questionnaire Response 265 TABLE 4. Estimated costs* of including a pencil (vs. no pencil) in a mailing to nonrespondents in a hypothetical study with up to three mailings of a questionnaire to 10,000 people Study phase Pencil No. % Cost ($US) No pencil No. % Cost ($US) Mailing 1 (no incentive) Mailing 10,000 19,000 10,000 19,000 Completed questionnaire 6,900 69y 8,832 6,900 69y 8,832 Mailing 2 (pencil vs. no pencil) Mailing 3,100 6,448 3,100 5,890 Completed questionnaire 1,519 49y 1, y 1,270 Mailing 3 (no incentive) Mailing 1,581 3,003 2,108 4,005 Completed questionnaire y y 701 Mailings 1 3 combined Completed questionnaire 8, ,753 8, ,698 * Costs per person (from study 2): questionnaire mailing, $1.90 ($0.78 for the 16-page questionnaire, a 1-page letter, and envelopes þ $0.89 for packet assembly þ $0.23 for bulk mail postage); pencil, $0.18 ($0.16 for pencil þ no extra charge for assembly þ $0.02 for nonmachine-reading postage fee þ no extra postage); return postage, $1.28. y Response rate from that mailing, estimated as 69% for mailing 1 (average of studies 1 and 2), 49% for pencil and 32% for no pencil for mailing 2 (average of studies 1 and 2), and 26% for mailing 3 (from study 2). points, on average (3, 4). More recently, Edwards et al. (2) reviewed 45 studies of nonmonetary incentives and reported that such incentives modestly increase response rates, on average (odds ratio ¼ 1.19, 95 percent confidence interval: 1.11, 1.28; approximately a 5 percentage point increase for response rates in the range of percent). The nonmonetary incentives included pencils, pens, lottery tickets, coffee coupons, tea bags, and newspaper articles on the study. Our observed effect was much greater: 15 percentage points in one study and 19 percentage points in the other. This difference may result from our trials being limited to nonresponders to a first mailing. Alternatively, we may have observed a stronger effect than the average effect in the other nonmonetary incentive trials because a pen or pencil is a facilitating factor as well as a token of appreciation. It may facilitate completing the questionnaire because the respondent does not have to search for a pen or pencil before beginning to fill out the survey, or perhaps some homes do not have a pen or pencil available. However, as we noted above, the trials most similar to ours did not find a significant effect of including a pen or pencil. Advantages of this report are that it includes two independent trials, one for population-based study recruitment and one for a follow-up survey in a cohort study. Limitations are that both trials were based on lists of people who had participated previously in some way (persons who had undergone mammograms in New Hampshire and agreed to be contacted for research and persons who had completed a cohort baseline questionnaire 2 years earlier in the VITAL Study). Therefore, our response rates to the first mailing were high, and our results for the second mailing may not be generalizable to studies that have low response rates to a first mailing. Another limitation is that we only tested inclusion of a pen/ pencil in the second mailing. Our rationale was that it would be too expensive to include a pen/pencil in the first mailing, which was to be sent to tens of thousands of people in the full studies. We also felt that a large percentage of those contacted would participate without an incentive, so the costs of the pen/pencil could be reserved for the nonrespondents. Our studies cannot answer the question of whether inclusion of a pen/pencil in a first mailing would increase response rates. Finally, our estimates of costs were specific to our findings on the effect of a pen/pencil on response rates and to the associated costs in one of our studies. Therefore, researchers may wish to conduct their own preliminary trial and cost estimation across all mailings before deciding whether to include a pen or pencil in some mailings. In summary, this study suggests that inclusion of a studylogo pen or pencil in a second questionnaire mailing to nonrespondents is a cost-effective method of increasing response rates. Further studies of the effect on response rates and associated costs of including a pen or pencil in a first mailing are needed to understand whether that would be a more or less cost-effective approach. ACKNOWLEDGMENTS This research was supported by National Cancer Institute grants R01 CA74846 and R01 CA Conflict of interest: none declared.

6 266 White et al. REFERENCES 1. Armstrong BK, White E, Saracci R. Principles of exposure measurement in epidemiology. (Monographs in epidemiology and biostatistics, vol 21). New York, NY: Oxford University Press, Edwards P, Roberts I, Clarke M, et al. Increasing response rates to postal questionnaires: systematic review. BMJ 2002; 324: Yammarino FJ, Skinner SJ, Childers TL. Understanding mail survey response behavior: a meta-analysis. Public Opin Q 1991;55: Church AH. Estimating the effect of incentives on mail survey response rates: a meta-analysis. Public Opin Q 1993;57: Aveyard P, Manaseki S, Griffin C. The cost-effectiveness of including pencils and erasers with self-completion epidemiological questionnaires. Public Health 2001;115: Hoffman SC, Burke AE, Helzlsouer KJ, et al. Controlled trial of the effect of length, incentives, and follow-up techniques on response to a mailed questionnaire. Am J Epidemiol 1998;148: Etter J-F, Perneger TV, Ronchi A. Collecting saliva samples by mail. Am J Epidemiol 1998;147: Carney PA, Poplack SP, Wells WA, et al. Development and design of a population-based mammography registry: The New Hampshire Mammography Network. Am J Roentgenol 1996;167: White E, Patterson RE, Kristal AR, et al. VITamins And Lifestyle cohort study: study design and characteristics of supplement users. Am J Epidemiol 2004;159: Clark TJ, Khan KS, Gupta JK. Provision of pen along with questionnaire does not increase the response rate to a postal survey: a randomised controlled trial. J Epidemiol Community Health 2001;55: Willimack DK, Schuman H, Pennell B-E, et al. Effects of a prepaid nonmonetary incentive on response rates and response quality in a face-to-face survey. Public Opin Q 1995;59: