Nonresponse Bias in a Longitudinal Follow-up to a Random- Digit Dial Survey

Transcription

1 Nonresponse Bias in a Longitudinal Follow-up to a Random- Digit Dial Survey Doug Currivan and Lisa Carley-Baxter RTI International This paper was prepared for the Method of Longitudinal Surveys conference, July at the University of Essex, Colchester, England. The authors gratefully acknowledge Joanne Pais of RTI for programming the analyses presented here; Barbara Bibb of RTI for technical assistance with the data file; and project director Matthew Farrelly of RTI and the New York State Department of Health for supporting this research. All conclusions and opinions expressed in this paper are solely those of the authors.

2 Nonresponse Bias in a Longitudinal Follow-up to a Random-Digit Dial Survey Abstract In an era of decreasing response rates to most social surveys, researchers are increasingly concerned that nonrespondents will differ from respondents in ways that directly impact survey estimates. The same consideration applies to panel attrition in longitudinal surveys. Losing a significant portion of sample members across data collection waves can produce nonresponse bias in subsequent estimates, if the nonrespondents differ in relevant ways from those who continue to participate. This paper examines nonresponse bias in a longitudinal follow-up survey to a random-digit dial telephone survey in the state of New York. To assess potential nonresponse bias in the follow-up survey with eligible sample members, we examined relationships among baseline survey experiences, smoking behavior and other health indicators, respondent characteristics, and follow-up survey outcomes. The analysis included (1) examination of baseline factors associated with participating in the follow-up survey and (2) assessment of how follow-up nonresponse affected key survey estimates, such as health status, smoking behavior, and beliefs about smoking and health. Multivariate analysis found some baseline survey experiences, smoking behaviors, and respondent characteristics predicted follow-up survey outcomes. Bivariate comparisons revealed only a few significant differences in key survey measures between the full sample of eligible first wave respondents and the smaller set of second wave participants. These findings indicate prior survey experiences and respondent characteristics can be related to participation in subsequent waves, although the impact on nonresponse bias was quite modest. We discuss these results in terms of sample members interest in continuing participation, the use of incentive protocols in longitudinal surveys, and the potential for nonresponse bias.

3 Introduction A fundamental concern of survey researchers is that nonrespondents often differ from respondents in systematic ways that can affect survey estimates. This concern assumes that one or more key differences between respondents and non-respondents may be related to survey indicators, thereby creating the potential for nonresponse bias. The same consideration applies to panel attrition in longitudinal surveys. Losing a significant portion of sample members across data collection waves can produce nonresponse bias in subsequent estimates, if the nonrespondents differ in relevant ways from those who continue to participate. For example, if sample members who leave a panel study of health issues do so because of poor health, this attrition could directly affect future estimates of health status in the target population. Several analyses of panel attrition can be found in the literature, especially studies evaluating large panel surveys in the United States, such as the Panel Study of Income Dynamics (Lillard and Panis, 1998;Zabel, 1998), the Survey of Income and Program Participation (Zabel, 1998), and the National Election Studies panel (Lepkowski and Couper, 2002). In addition, other research has looked at survey design features influencing panel attrition in large panel studies such as the British Household Panel Study (Laurie, Smith, and Scott, 1999). These analyses have provided useful models of the attrition process and identified factors related to panel nonresponse. Zabel (1998) also assessed the impact of attrition on one key survey estimate, labor market attachment, but the primary focus of these studies was analyzing how different factors predict attrition. This information is quite useful for predicting attrition and understanding the attrition process, but does not provide direct evidence on the impact of nonresponse on bias in panel survey estimates. The fact that attrition is relatively low in these large, in-person panel studies may be one explanation for the limited focus on nonresponse bias in these panel surveys. With relatively low attrition, the potential for nonresponse bias is clearly limited. Conversely, in studies with more challenging and less successful panel maintenance, nonresponse bias may more important to explore. For example, panel members who are originally recruited by telephone sampling methods such as random-digit-dialing (RDD) are likely to have less identification with the study than those recruited face-to-face. Existing analyses of panel attrition have focused almost exclusively on in-person surveys, which use area probability sampling or similar methods. Respondents recruited by telephone likely have less motivation to participate in follow-up waves, especially the second wave. To date, no published studies appear to have examined panel attrition and nonresponse bias in studies where the initial survey wave involved RDD sampling for telephone interviews. This paper examines nonresponse bias in a longitudinal follow-up survey to a random-digit dial (RDD) telephone survey, the New York Adult Tobacco Survey (NY ATS). The NY ATS and its follow-up survey, the New York Adult Cohort Survey (NY ACS) focus 1

4 on tobacco use, smoking behavior, and related health issues among residents of the state of New York in the United States. To assess potential nonresponse bias in the follow-up survey with eligible NY ATS participants, we examine relationships among baseline survey experiences, smoking behavior and other health indicators, respondent characteristics, and follow-up survey outcomes in the NY ACS. Our analysis is comprised of two parts (1) examination of baseline factors associated with participating in the follow-up survey and (2) direct assessment of how follow-up nonresponse affected key survey estimates, such as health status, smoking behavior, and beliefs about smoking and health. Combined, these analyses provide a complete picture of nonresponse bias in a telephone follow-up survey. Survey Design and Panel Attrition Researchers have identified a number of survey design features related to panel attrition in longitudinal surveys. Number of waves. Increasing the number of survey waves tends to produce greater cumulative nonresponse, as more panel members become difficult to locate or discontinue participation for other reasons (Kalton and Citro, 1993; Lepkowski and Couper, 2002). Frequency of data collection. More frequent data collection can be associated with panel fatigue, whereby panel members attrit because of the high burden of participation (Kalton and Citro, 1993; Lepkowski and Couper, 2002). Length of time between data collection waves. Longer time intervals between survey waves generally increases the risk that panel members will move and not be traceable for subsequent waves (Lepkowski and Couper, 2002). Contact between data collection waves. Contact between survey waves increases the likelihood of being able to find panel members in subsequent waves, especially when confirming/updating contact information is part of the contact process (Lepkowski and Couper, 2002; Laurie, Smith, and Scott, 1999). Use of incentives. Incentives, including money and gifts, are frequently used and found to limit panel attrition, although research to date provides few published empirical assessments of the impact of incentives on nonresponse in longitudinal studies (Kalton and Citro, 1993; Laurie, Smith, and Scott, 1999; Singer, 2002). Singer (2002) review of unpublished studies indicates that larger incentive payments (such as $20) have been effective in maintaining panel participation, but somewhat smaller amounts ($10) have not been shown to significantly improve retention compared to no incentive. Respondent eligibility rules. How eligibility for subsequent waves is determined in longitudinal surveys can increase the challenge to sample maintenance. For 2

5 example, panel surveys that pursue both prior-wave respondents and nonrespondents are likely to have greater attrition issues in a subsequent wave than those surveys that only include prior-wave respondents. (Kalton and Citro, 1993; Lepkowski and Couper, 2002). Sample design. If the sample design for a panel survey includes a relatively high proportion of individuals who are more difficult to include in surveys, than panel attrition is likely to be greater (Kalton, Kaspryzk, and McMillen, 1989). For example, a longitudinal study may oversample low-income members of the target population, who tend to be more transient and are therefore more difficult to track. Because individual studies combine these design features in different ways, the impact of survey design on panel nonresponse is probably best characterized as the combined effects of these factors. Panel Nonresponse and Bias Our concern with how study design features relate to panel attrition is only important to the extent that it can improve our understanding of the association between survey outcomes and nonresponse bias. The primary concern is that panel nonrespondents may differ in systematic ways from panel members (Kalton, Kaspryzk, and McMillen, 1989). As in cross-sectional surveys, nonresponse bias in panel surveys is a combination of the rate of the rate of nonresponse and the difference between participants and non-participants. If panel attrition is low but nonresponders differ greatly from responders, nonresponse bias may still be observed in some survey measures. Conversely, panel attrition may be relatively high but nonresponders may not differ at all from responders, so that nonresponse bias is minimal or nonexistent in the survey statistics. One model of panel attrition that attempts to link panel survey features and potential nonresponse bias is Lepkowski and Couper s (2002) theory of nonresponse in panel surveys. Their preliminary theory attempted not only to identify key factors associated with panel nonresponse, but also how these different factors relate to specific types of nonresponse. The different types of nonresponse they considered are inability to locate panel members, inability to contact sample members, and non-completion of the next survey wave (Lepkowski and Couper, 2002, p.261). By dividing panel attrition into its multiple sources, this theory allows for more precise understanding of how different factors may lead to nonresponse bias. If we can identify the characteristics of panel members who are more 3

6 difficult to locate in subsequent survey waves, for example, then we can understand the potential kinds of nonresponse bias this attrition may produce. 1 Lepkowski and Couper (2002) provided two empirical examples of the impact of survey design features and respondent/household characteristics on location and completion in the second wave of a panel study. Their analysis showed that survey design and respondent characteristics did affect location and cooperation propensity somewhat differently, as they anticipated. In the first study they examined, location propensity was associated with socio-demographic and geographic measures. Several demographic factors including are, gender, race, and education as well as geographic region were significantly related to locating sample members at wave 2. Virtually none of these factors was significantly associated with cooperation with the wave 2 interview, once located. Cooperation with the interview was related more to community attachment and social integration, including measures such as renting versus owning a home, formal social integration, and volunteering activities. In the second study they examined, sociodemographic and geographic factors were not as predictive of location propensity. Instead, the most important factors are experience in the first wave and willingness to be found. Lepkowski and Couper s (2002) work represents an important advance for understanding potential sources of nonresponse bias in longitudinal studies. At the same time, their research did not also include an actual analysis of nonresponse bias in key measures for either of the surveys they examined. Research Questions This paper extends existing work on nonresponse bias in panel surveys by examining both the sources and impact of nonresponse in the follow-up wave of the NY ATS, the New York Adult Cohort Survey (NY ACS). We investigate two sets of research questions to understand the nature of potential nonresponse bias in the NY ACS. The first set of questions concerns the impact of baseline survey conditions and respondent characteristics on participation in the follow-up survey, while the second set of questions concerns the specific impact of nonresponse to the follow-up survey on key survey estimates. First, we inquire how survey experiences, health status, and respondent characteristics from the first survey wave might be related to participation in the second wave. Because we are interested in all sources of panel nonresponse, we examine three elements of follow-up survey response: 1 Lynn, Clarke, Martin, and Sturgis (2002) make a similar distinction for cross-sectional surveys, where they examine ease of contact and reluctance to participate as separate survey outcomes. Across different surveys, they found both dimensions to be systematically associated to substantive and demographic survey items. 4

7 (1) whether panel members were located for the follow-up survey, (2) whether panel members refused to participate in the follow-up survey, and (3) whether panel members completed the follow-up survey. Although Lepkowksi and Couper (2002) initially distinguished among three components of panel survey participation location, contact, and completion they dropped the dimension of contact from their analyses. We also exclude contact from consideration, since location and contact in telephone surveys are nearly synonymous. Surveys rarely produce situations where we confirm location of sample members without some form of contact with them. Because we are interested in the major components of panel nonresponse, we expand the two dimensions of participation to include a third element, refusal to the follow-up survey. We seek to ascertain which survey and respondent factors affect these three response outcomes in the follow-up survey, and whether these factors have different effects on location, refusal, and completion. The second question we consider is to what extent nonresponse to the follow-up survey affects estimates of key survey measures of tobacco-related behaviors and attitudes. Key measures in the NY ACS include current smoking status, attempts to quit smoking, and beliefs about the harm of smoking to health. A further question is how wave nonresponse affects the distribution of respondents across characteristics such as age, education, and household type. Although we are primarily interested in the impact of nonresponse on the central survey measures, examination of the impact of nonresponse on the characteristics of follow-up participants may indicate further ways that nonresponse affects follow-up survey estimates. Since we have data for the complete set of first wave participants eligible for the follow-up survey, we can examine how the second wave sample compares to the second wave respondents on these measures (Menard, 1991). Methods Surveys. The New York Adult Tobacco Survey (NY ATS) is a statewide household survey of New York residents aged 18 and older that began with the third quarter of This survey measures the prevalence of current smoking, smoking and smoking cessation behaviors, and attitudes and beliefs about tobacco use and tobacco products. The quarterly surveys are designed to provide timely surveillance and evaluation data to inform the New York State Tobacco Control Program. The NY ATS sample follows a stratified dual frame design, with sampled telephone numbers drawn equally from (1) a random-digit-dial (RDD) frame and (2) a residential directory-listed frame. This design provides a representative sample while increasing the incidence rate of current residential units to improve data collection efficiency. Both sample frames are stratified across eight county groups across the state, producing a sample with 16 total strata. Within each household associated with the 5

8 sampled numbers, one adult age 18 or older is selected. The CATI selection protocol also oversampled current smokers by selecting 80% of current smokers identified during the initial household listing of adults and smokers. The sample design, selection procedures, and weighting adjustments permit the calculation of precise statewide and regional estimates. The NY ACS is a longitudinal follow-up survey to NY ATS. The NY ACS has a number of important design features that may be relevant to the present research: The survey attempts to reinterview NY ATS respondents in the same quarter one year later than the first wave interview. No contact with sample members is made between waves, unless NY ATS respondents happen to call the toll-free line associated with the study to ask questions or provide information. Interviewers do not initially offer NY ATS and NY ACS sample members an incentive to participate. For both waves, when selected sample members or other members of the household refuse the initial survey request, the case is defined as a refusal. The refusal conversion protocol offers initial refusers a $20 incentive, sent to the respondent as a check. NY ATS sample members are eligible for selection to the NY ACS sample if the following conditions are met: (1) The sample member completes the baseline NY ATS survey. (2) The sample member indicates in the NY ATS interview that they currently smoke or have quit smoking in the past 12 months. (3) The sample member agree to provide information to be contacted for future survey waves. The present study uses NY ATS data from quarter 4 of 2003 and the NY ACS followup in quarter 4 of A total of 2,063 New York residents completed interviews in quarter 4 of The AAPOR response rate 3 for this quarter of the NY ATS was 23.8%. The average interview length was 26.7 minutes. On average, the survey was about 6 minutes longer for current smokers (31 minutes) compared to non-smokers (25 minutes). The difference in interview length is comprised of the additional questions on smoking behavior, including cigarette usage, purchasing cigarettes, and attempts to quit smoking. The average time for those quarter participants who were eligible for the NY ACS follow-up was also 31 minutes, since all of these sample members were current smokers or has recently quit smoking. 2 Although we examine only one quarterly data set of first and second-wave participants in this paper, both surveys include additional quarterly data sets. Data collection procedures, response rates, and sample member characteristics have remained quite stable over the course of subsequent quarters. We will add subsequent quarters to this paper to expand our research in the future to confirm the patterns found in this paper. 6

9 Table 1. Key Survey Outcomes for the Wave 1 New York ATS and Wave 2 New York ACS (unweighted) Survey Outcomes NY ATS Respondents Eligible for the NY ACS (n = 494) Wave 1 - NY ATS Initially refused participation in the NY ATS % Ten or more calls to complete the NY ATS % Interview NY ATS Interview lasted 30 minutes or % longer Wave 2 - NY ACS Located for NY ACS follow-up % Refused NY ACS interview % Completed NY ACS interview % Table 1 presents a summary of survey outcomes for the wave 1 NY ATS and wave 2 NY ACS. Among the 2,063 NY ATS respondents from quarter 4 of 2003, 494 were current smokers or recent quitters who agreed to future contact. The majority of these sample members, 437 or 88%, were currently smoking when interviewed in wave one. The other 57 (12%) eligible participants reported quitting smoking 12 months or more recently prior to the interview. We then attempted to re-contact these 494 sample members via mailed lead letters and telephone calls in the fourth quarter of Those contacted were invited to complete a follow-up telephone interview. A total of 353 sample members (78%) were located for the follow-up wave. Of these panel members, 246 (70%) completed the NY ACS interview, for an overall AAPOR response rate 3 of 50%. The average interview length was 29.3 minutes. Among the other ample members located for the second wave, 71 refused to complete the interview and the remaining 36 were not able to complete the interview due to time limitations or other constraints. 7

10 Measures. The appendix presents all survey measures from quarter 4 of the 2003 NY ATS and quarter 4 of the 2004 NY ACS used in the analyses. These measures include survey process and outcome data, key indicators of tobacco use and health, and demographic characteristics from the two survey waves. The survey items on tobacco use and health include both behavioral measures, such as smoking and quit attempts, as well as attitudinal measures, such beliefs on the impact of second-hand smoke on health. Analysis. To address the two main research questions, the analysis was divided into two parts. The first part followed Lepkowski and Couper s (2002) approach to examine factors related to participation in the NY ACS. This analysis included three survey outcomes location, refusal, and completion. We considered three main types of variables measured in the baseline NY ATS survey that might be associated with the three follow-up survey outcomes: Survey experience: Variables such as initial refusal to participate, the number of calls to complete the interview, and the length of the interview may be related to sample members willingness to participate in the follow-up survey. Smoking and health status: Factors such as smoking status, physical health problems, and mental health issues may also affect sample members willingness or ability to participate in the second wave of the study. Demographic characteristics: Respondent characteristics such as age, race/ethnicity, employment status, education, and others may also be related to our ability to include sample members in the NY ACS follow-up. We chose specific NY ATS measures to represent some of the key factors that might have influenced wave 2 participation and/or might have produced nonresponse bias in wave 2. These models most closely follow Lepkowski and Couper s (2002) approach, but we did not have all of the same measures from their survey and we added variables important to the NY ATS. Furthermore, because we wanted to be able to directly compare how these different factors were associated with location, refusal, and completion in the second wave, this analysis included the same set of predictors in three logit regression models. As Lepkowski and Couper s (2002) argued, and observed in one study, different predictors may be more strongly or weakly related to the three different survey outcomes. This first part of the analysis provided information on what factors were related to participation in the second wave of the study, and exactly how they were related to the three different dimensions of response. To take the analysis of panel attrition and nonresponse bias a step further, the second part of the analysis examined more directly how nonresponse to the follow-up affects key survey estimates. The approach followed for this part of the analysis takes advantage of the fact that we have pertinent survey estimates on the full sample for the second wave of the 8

11 study, the 494 NY ATS respondents who were current smokers or recent quitters. We used t- tests to compare several key variables found in both quarter 4 of the 2003 NY ATS and quarter 4 of the 2004 NY ACS data sets. These measures were generally classified into four types: (1) smoking behaviors, including tobacco use and smoking cessation, (2) beliefs about the impact of smoking on health, (3) physical and mental health status, and (4) demographic characteristics. For this analysis, we chose a range of measures reflecting key substantive and demographic items from the NY ATS and ACS. Measures were chosen to represent the major variables assessed in the studies, and were not intended to be exhaustive. Significant differences between the full sample of eligible panel members and follow-up participants provided evidence on the extent of bias in these key survey measures attributable to nonresponse. Results 1 Wave 1 Predictors of Wave 2 Survey Outcomes The first part of the analysis assessed whether survey experience, health status, or respondent characteristics measured in the NY ATS were associated with three dimensions of participation in the NY ACS location, refusal, and completion. Table 2 presents the results of logit regression analysis of these three survey outcomes on the follow-up survey on these first wave measures. 3 We present only the full models, to observe which NY ATS variables have significant associations with NY ACS outcomes, controlling for the other variables. Starting with location, we found one measure of survey experience and a few demographic characteristics were associated with this survey outcome. NY ATS participants who required 10 or more telephone calls to complete the first wave interview were less likely to be located for the second wave NY ACS, although the association was marginal (p = 0.087). This finding suggests two conclusions. First, locating sample members primarily through telephone data collection can be challenging. Researchers may not be able to confirm location of those panel members who do not usually answer calls from unknown persons or organizations, as well as those who are intentionally avoiding such calls. Second, sample members who had negative reactions to being asked to participate in the first survey wave may be more difficult to locate. Lepkowski and Couper (2002) found those who thought the baseline interview was too long or refused to answer a high number of questions were less likely to be located at wave 2. These findings provide further evidence that prior wave survey experiences may be related to location or contact in later waves. 3 All analyses are based on data adjusted using base weights, which do not include adjustments for nonresponse nor post-stratification. 9

12 Among the demographic items, we found that sample members age 45 and over were much more likely than younger sample members to be located for the NY ACS. We also found that sample members who had at least one child in their household were more likely to be located for the second wave. Similarly, Zabel (1998) observed that panel members with more children were less likely to become nonresponders across two different studies. NY ACS sample members with relatively lower incomes were less likely to be located, consistent with Laurie, et al (1999). All three of this findings are likely related to sample member life cycles and mobility. Greater mobility among younger members of the population was a likely factor in increasing the difficulty of tracing them. Both older sample members and those with children were likely less mobile and therefore easier to find for the second wave. Laurie, et al (1999) found that sample members who did not have any children in the first wave of their study were more likely to be nonrespondents at later waves. Interestingly, Lepkowski and Couper (2002), who also looked directly at location in the second wave of a survey, found a different set of demographic factors were related to location. These authors observed significant positive associations with wave 2 location with both higher education and being currently employed. Male sample members also appeared to be more difficult to find for the follow-up wave, although this association was of marginal significance (p = 0.090). Turning to refusals at the second wave, we found first wave interview length, smoking status, and income. NY ATS respondents whose interview lasted 30 minutes or longer appeared to be more likely to refuse to participate in the NY ACS follow-up, but this association was somewhat weak (p = 0.080). This finding may be the consequence of greater perceived and real burden for these participants, and they are consistent with Laurie, et al s (1999) findings. A similar argument can be made for those who were current smokers in the first wave, who were also more likely to refuse to participate in the second wave. As noted above, the NY ATS interview was about 6 minutes longer for smokers. Interestingly, Zabel (1998) observed the opposite relationship among participants in face-to-face surveys: those with longer prior interviews were actually more likely to continue in subsequent panel waves. Zabel suggested this finding may be explained by the fact that those who have longer interviews may have developed better rapport with their interviewers, which in turn increased their attachment to the study. At the same time, Laurie, et al (1999) found no such evidence in their analysis of an in-person panel survey. Only one demographic characteristic, income level, was associated with refusal to the NY ACS. Sample members with lower incomes were more likely to refuse to participate than those with higher incomes. No other demographic items were associated with refusal to the follow-up survey. 10

13 Table 2. Odds Ratios from Logit Regression of Wave 2 New York ACS Survey Outcomes on Wave 1 New York ATS Measures (n = 494) New York ACS Follow-up Outcomes New York ATS Measures Location Refusal Completion Survey Experiences Initially refused participation or more calls to complete interview * Interview lasted 30 minutes or longer Smoking and Health Status Current smoker ** Recent physical health problems * Recent mental health problems Demographic Characteristics Age 45 or over 4.768** ** Male Hispanic/Latino White Married One or more children in household 2.274* College or higher degree Currently employed Income below $50, * 0.426* p <.10 * p <.05 ** p <.01 11

14 The final survey outcome in the NY ACS, actual completion of the interview, was associated with only a few factors, and most of them survey experiences. Those who required 10 calls or more to complete the NY ATS and those for whom the NY ATS interview lasted 30 minutes or longer were less likely to complete the NY ACS interview. These results were consistent with Lepkowski and Couper s (2002) findings on the impact of prior survey experiences with later cooperation. They found those who enjoyed the baseline interview were more likely to cooperate in the follow-up, but those who thought the baseline interview was repetitive were less likely to cooperate in the follow-up. Two other factors had marginally significant associations with completing the second wave interview were recent physical health problems and age. Sample members who reported in the NY ATS that they had recent physical health problems were actually more likely to complete the NY ACS interview. This finding seems counter-intuitive, but may be explained by the greater ease of contacting sample members who are frequently at home because of health problems. Following the findings for location, sample members age 45 and older were more likely to complete the follow-up interview. Lepkowski and Couper (2002) found a similar effect of age on survey cooperation in the second study they examined. Results 2 Comparison of Survey Measures for Full Sample and Wave 2 Respondents The second part of the analysis was designed to assess nonresponse bias by comparing key survey measures from the full sample of NY ATS participants eligible for the NY ACS and those sample members who actually completed the NY ACS follow-up interview. We chose two smoking behaviors indicators, five items on beliefs about smoking and health, four measures on health status, and nine demographic indicators, for a total of 20 individual survey items. These items represent a sampling of survey measures across key areas from the NY ATS and ACS, but were not meant to reflect the complete range of survey items included in the two surveys. Table 3 present the results from two-sample t-tests. Overall, the t-test comparisons between the 494 eligible NY ATS participants and the 246 actual NY ACS respondents revealed only three statistically significant differences in survey estimates. Among the 20 measures compared, beliefs about the impact of smoking on lung cancer, recent physical health problems, and age were the only variables that differed between the two samples. NY ACS respondents were less likely to believe that smoking causes lung cancer compared to those who did not complete the follow-up interview. Following the results from the first part of the analysis, sample members who reported in the NY ATS that they had recent physical health problems were actually more likely to complete the NY ACS interview. The third significant finding was also consistent with the results from part 1: sample members 45 and over were more likely to be represented among the NY 12

15 Table 3. Comparison of Key Measures between Eligible Wave 1 New York3 ATS Participants and Wave 2 New York ACS Respondents Survey Measures Eligible Wave 1 NY ATS Participants (n = 494) NY ACS Wave 2 Respondents (n = 246) Smoking Behaviors Current Smoker 85.6% 80.4% Attempted to quit in past year # 40.7% 36.5% Beliefs about Smoking and Health Smoking has already affected health 73.4% 67.9% Second-hand smoke causes lung cancer 77.4% 65.2% * Second-hand smoke causes heart 68.0% 61.9% disease Second-hand smoke causes respiratory 88.2% 88.0% problems in children Favors state-wide ban on smoking in 32.9% 42.4% public places Health Status Seen doctor in past year 61.3% 67.4% Fair or poor overall health 17.6% 22.4% Recent physical health problems 15.8% 26.7% * Recent mental health problems 34.9% 37.0% Demographic Characteristics Age 45 or over 30.9% 44.8% * Male 46.8% 41.6% Hispanic/Latino 12.1% 12.8% White 68.9% 74.2% Married 38.3% 44.4% One or more children in household 46.3% 47.2% College or higher degree 8.7% 13.0% Currently employed 55.7% 58.6% Income below $50, % 58.8% # Measured only among current smokers in each wave (NY ATS n = 434, NY ACS n = 213) Difference between eligible NY ATS participants and NY ACS respondents is significant at: + p <.10 * p <.05 ** p <.01 13

16 ACS respondents. None of the other measures indicated statistically significant differences between those who did and did not complete the NY ACS interview. Several other measures appeared to show minor differences in measures between the two survey waves, but none of these differences was statistically significant. For example, both the proportion of smokers and those smokers who attempted to quit in the past year were slightly lower among the NY ACS participants. Also, on each of the first three measures of beliefs about smoking and health, NY ACS respondents appeared to be slightly less concerned about the dangers of tobacco smoke for health. Among demographic indicators, second wave responders appeared to be slightly less likely to be male, slightly more likely to be white, slightly more likely to be married, slightly more likely to have a college degree, slightly more likely to be employed, and slightly less likely to have incomes below $50,000. Again, none of these differences was statistically significant at conventional levels. These findings may be partly the results of a lack of statistical power in the analysis. These nonsignificant differences are as high as 10%. The overall pattern in the data may suggest some practical implications for similar surveys. For example, the first and second parts of the analysis combined suggest a number of subgroups that may be more difficult to include in this kind of follow-up survey younger respondents, members of certain racial/ethnic groups, males, those who are single, those who are not employed, and those with lower incomes. Discussion The two parts of our analyses of the NY ATS and ACS data present an interesting picture of nonresponse bias in a follow-up survey with RDD participants. First, we found several important factors from the first wave including survey experiences, smoking and health status, and demographic characteristics were predictive of participation in the second wave along three dimensions. Furthermore, the same set of predictors had varying degrees of association with different dimensions of participation, just as Lepkowski and Couper (2002) had originally hypothesized. Second, despite the impact of these factors on follow-up survey participation, we observed relatively little significant evidence of nonresponse bias in the second wave among 20 survey measures. Only three survey estimates changed significantly from the first to second waves of the study. Our findings have a few important implications for nonresponse bias in longitudinal surveys and particularly follow-up surveys with RDD participants. First, we observed that survey experiences did have some impact on all three dimensions of second wave participation location, refusal, and completion. Consistent with the literature on panel studies, survey design features did make a difference in the response process to subsequent waves (Kalton and Citro, 1993; Kalton, Kaspryzk, and McMillen, 1989; Laurie, Smith, and Scott, 1999; Lepkowski and Couper, 2002). Respondents who had negative reactions to prior 14

17 waves of the survey such as refusing to participate, delaying participation across numerous contact attempts, refusing to answer multiple questions, or having a relatively long interview tend to be less likely to participate in subsequent waves. Furthermore, following up with survey participants who were initially more likely to refuse or delay participation increases the challenge of achieving high participation rates. NY ATS participants selected for the NY ACS not only had to complete a longer interview, on average, than other respondents, but the additional questions they answered focused on their current or recent smokers. Many of these sample members may have limited interest to complete a follow-up survey of similar length and content, given the increasing limitations on tobacco use in their home state. Relevant evidence for this conclusion is Lepkowski and Couper s (2002) finding that sample members with greater knowledge and interest in the survey topic were more likely to cooperate in the second wave of one of the studies they examined. 4 Any association between sample members interest in the survey topic and their decision to participate in the study raises concerns about the potential for nonresponse bias. A related issue in the survey design of NY ATS and ACS is the use of incentives. Sample members who initially refuse to complete both surveys are then offered an incentive of $20. Current smokers are overrepresented among those who refuse and, consequently, among those who complete the interview only after being offered the incentive. One concern in the use of such incentives is that those respondents with little interest in the survey topic might also be less motivated to provide complete and accurate data than other sample members (Shettle and Mooney, 1999). When offered a monetary incentive, these sample members may simply provide a minimum level of effort in completing the survey. Singer s (2002) review of the available literature suggests respondents who are paid an incentive do not generally provide lower quality data. Still, when payment of an incentive is associated with a key survey measure, researchers might reasonably be more concerned about the impact of both unit and item nonresponse on data quality. One limitation of the NY ACS survey design is that the study does not include specific activities to contact sample members between waves. We could therefore not assess the impact of this common panel survey design element on nonresponse to the follow-up. Adding these kinds of activities could provide a means to gain insight on the nature of second-wave nonresponse, especially with respect to locating issues. Despite some evidence of differential propensity to among certain NY ATS sample members to participate in the NY ACS, we found limited evidence of nonresponse bias. This is an interesting finding, given that the (unweighted) second wave response rate did not quite reach 50%. Our results are in this way similar to recent studies such as Keeter, Miller, 4 Looking at several cross-sectional surveys, Groves, Presser, and Dipko (2004) also found some evidence that sample member interest was significantly related to participation. 15

18 Kohut, Groves, and Presser (2000) that find low response rates did not translate into observable nonresponse bias. One possible explanation for the minimal impact of panel nonresponse on follow-up survey estimates may be related to the fact that the NY ACS panel was limited to only a particular subset of NY ATS participants. Only those who currently smoked or had recently quit smoking were included in the panel. This special subgroup likely had a more homogeneous set of experiences and views compared to the full set of all eligible NY ATS participants. If this was the case, considerable nonresponse to the follow-up may not have been sufficient to produce significant differences in survey estimates. If the data were collected among a wider set of NY ATS participants, analysis of nonresponse bias among a more heterogeneous set of survey respondents might have produced greater differences between the first wave and second wave data. This conclusion points us to the need for further research on the conditions under which panel attrition, of varying degrees, does or does not produce observable nonresponse bias. 16

19 References Groves, R.M., Presser, S. and Dipko, (2004). The Role of Topic Interest in Survey Participation Decisions. Public Opinion Quarterly, 68, Kalton, G. and Citro, C. (1993). Panel Surveys: Adding the Fourth Dimension Survey Methodology 19(2): Kalton, G., Kasprzyk, D., and McMillen, D. (1989). Nonsampling Error in Panel Surveys in D. Kasprzyk, et al. (Eds.) Panel Surveys. New York: Wiley. Keeter, S., Miller, C., Kohut, A., Groves, R., Presser, S. (2000). Consequences of Reducing Nonresponse in a National Telephone Survey, Public Opinion Quarterly, 64, Laurie, H., Smith, R., and Scott, L. (1999). Strategies for Reducing Nonresponse in a Longitudinal Panel Survey. Journal of Official Statistics, 15, Lepkowski, J. M. and Couper, M.P. (2002). Nonresponse in the second wave of longitudinal household surveys. In Survey Nonresponse, R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.A. Little (eds.), pp New York: Wiley. Lillard, L.A. and Panis, C.W.A. (1998). Panel Attrition from the Panel Study of Income Dynamics: Household Income, Marital Status, and Mortality. Journal of Human Resources, 33, Lynn, P., Clarke, P., Martin, J., and Sturgis, P. (2002). The Effects of Extended Interviewer Efforts on Nonresponse Bias. In Survey Nonresponse, in R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.A. Little (eds.), pp New York: Wiley. Menard, S. (1991). Longitudinal Research. Newbury Park, CA: Sage. Shettle, C. and Mooney, G. (1999). Monetary Incentives in U.S. Government Surveys. Journal of Official Statistics, 15, Singer, E The Use of Incentives to Reduce Nonresponse in Household Surveys. In Survey Nonresponse, in R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.A. Little (eds.), pp New York: Wiley. Zabel, J. (1998). An analysis of attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation, with an Application to a Model of Labor Market Behavior. Journal of Human Resources, 33,

20 Appendix: Survey Measures, Wording, Coding, and Frequencies from the 2003 New York Adult Tobacco Survey and 2004 New York Adult Cohort Survey Survey Experiences New York ATS/ACS Survey Item Coding Initially refused participation in NY ATS Ten or more calls to complete NY ATS interview NY ATS Interview lasted 30 minutes or longer n/a n/a n/a 0 = did not refuse initial survey request 1 = ever refused initial survey request 0 = 9 or fewer calls to completion 1 = 10 or more calls to completion 0 = interview length less than 30 minutes 1 = interview length 30 or more minutes Smoking Behaviors New York ATS/ACS Survey Item Coding Current smoker Attempted to quit in past year a. Have you smoked at least 100 cigarettes in your entire life? 1. Yes 2. No b. Do you now smoke cigarettes everyday, some days, or not at all? 1. everyday 2. some days 3. not at all During the past 12 months, have you stopped smoking for one day or longer because you were trying to quit smoking? 1. Yes 2. No (*Asked only of current smokers) 0 = non-smoker if a = 2 or (a = 1 and b = 3) 1 = current smoker if a = 1 and (b = 2 or b = 3) 0 = no 1 = yes 18

21 Beliefs about Smoking and Health Smoking has already affected health Second-hand smoke causes lung cancer Second-hand smoke causes heart disease Second-hand smoke causes respiratory problems in children Favors state-wide ban on smoking in public places New York ATS/ACS Survey Item Do you think that your smoking has already affected your health? 1. Yes 2. No Please tell me if you strongly agree, agree, disagree, or strongly disagree with the following statement: Smoking cigarettes causes lung cancer? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree Would you say that breathing smoke from other people s cigarettes causes heart disease? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree Would you say that breathing smoke from other people s cigarettes causes respiratory problems in children? 1. Strongly agree 2. Agree 3. Disagree 4. Strongly disagree Are you personally in favor, opposed to, or indifferent to the recently enacted New York State law prohibiting smoking in all public and work places, including bars and restaurants? 1. Favor 2. Oppose 3. Indifferent 4. Not familiar with law Coding 0 = no 1 = yes 0 = disagree or strongly disagree 1 = strongly agree or disagree 0 = disagree or strongly disagree 1 = strongly agree or disagree 0 = disagree or strongly disagree 1 = strongly agree or disagree 0 = oppose or indifferent 1 = favor 19

22 Health Status New York ATS/ACS Survey Item Coding Seen doctor in past year In the past 12 months, have you seen a doctor, nurse, or other health professional to get any kind of care for yourself? 1. Yes 2. No 0 = no 1 = yes Fair or poor overall health Recent physical health problems In general, would you say your health is: 1. Excellent 2. Very good 3. Good 4. Fair 5. Poor Now thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good? (0 30 days) 0 = excellent, very good, or good 1 = fair or poor 0 = 3 days or less 1 = more than 3 days Recent mental health problems Now thinking about your mental health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good? (0 30 days) 0 = 3 days or less 1 = more than 3 days Demographic Characteristics New York ATS/ACS Survey Item Coding Age 45 or over Male What is your age? (AGE IN YEARS) For survey purposes, I need to confirm if you are male or female? 1. Male 2. Female 0 = = = female 1 = male 20

23 Hispanic/Latino background White Are you Hispanic or Latino? 1. Yes 2. No Which one or more of the following would you say is your race: 1. White 2. Black or African American 3. Asian 4. Native Hawaiian or Pacific Islander 5. American Indian, Alaska Native 6. Other (specify) 0 = no 1 = yes 0 = race other than white 1 = white Married Are you: 1. Married 2. Divorced 3. Widowed 4. Separated 5. Never married, or 6. Living with a partner 0 = not currently married 1 = currently married (1) One or More Children in Household College or higher degree How many children live in your household who are younger than 5 years old? 5 through 11 years old? 12 to 17 years old? What is the highest level of school you completed or the highest degree you received? 1. Never attended school/only attended kindergarten 2. Grades 1 through 8 (Elementary) 3. Grades 9 through 12 (Some high school) 4. Grade 12 (High school graduate) 5. G.E.D 6. Some technical or vocational school 7. Some college, no degree 8. AA; technical or vocational school 9. AA; academic 10. BA, BS (College graduate) 11. At least some graduate or professional school 12. Graduate or professional degree 0 = no children in household 1 = at least one child in household 0 = less than college degree (< 10) 1 = college degree or higher (10 +) 21

24 Currently employed Are you currently 1. Employed for wages 2. Self-employed 3. Out of work for more than 1 year 4. Out of work for less than 1 year 5. Homemaker 6. Student 7. Retired 8. Unable to work 0 = not currently employed 1 = employed or self-employed (1 or 2) Income less than $50,000 Was your annual household income from all sources during 2002/2003 more or less than $50,000? 1. $50,000 OR MORE 2. LESS THAN $50,000 0 = income greater than $50,000 1 = income less than $50,000 22