The Future of Survey Research: Challenges and Opportunities

Transcription

1 The Future of Survey Research: Challenges and Opportunities Sponsored by The National Science Foundation Conference 1: October 3-4, 2012 Conference 2: November 8-9, 2012 Conference Organizers: Jon Krosnick Stanford University Stanley Presser University of Maryland Kaye Husbands Fealing University of Minnesota Steven Ruggles University of Minnesota Research Assistant: David Vannette Stanford University Hosted at the Hilton Arlington 950 North Stafford Street Arlington, Virginia 22203

2 Conference Overview The first portion of this two-part conference will focus on operational challenges and opportunities for survey researchers. The conference will begin with a new review of accumulated evidence about the widespread use of survey data and about the accuracy of those data. This evidence will set the tone for the conferences by making it clear that the method is both popular and reliable when implemented according to best practices. The conference will then turn to elucidating the many challenges and opportunities facing survey researchers seeking to do so. Outlining those challenges and opportunities will entail illuminating current knowledge about how best to cope with them and maintain data quality while preserving affordable costs. The net result of this conference will be (1) a set of insights about how surveys should be done today to maximize data quality (thereby specifying how major infrastructure surveys should be designed and carried out), (2) a list of the most important challenges and opportunities facing the methodology, and (3) a list of research questions that merit future investigation, perhaps with NSF grant support. 2

3 October 3 rd Agenda 7:30-8:30AM Continental breakfast 8:30-8:45AM Reasons For Optimism About the Accuracy of Survey Research Jon Krosnick Stanford University 8:45-9:40AM The Impact of Survey Nonresponse on Survey Accuracy Scott Keeter Pew Research Center Among the many challenges facing survey research, the rise of non-response is perhaps the most worrisome. Nonresponse does not automatically lead to biased estimates, but greater nonresponse means greater potential for bias. Accordingly, a significant body of research has been devoted to assessing the impact of nonresponse on the accuracy of survey estimates. My presentation will review this research and provide a summary of what is known about the accuracy of surveys, especially those in the social and political realms where the NSF has been an important source of support for research. I will also identify areas in which additional research on nonresponse is needed. 9:40-9:55AM Coffee Break 9:55-10:50AM Proxy Reporting Curtiss Cobb - GfK Many prominent studies, including the Current Population Survey (CPS) done by the U.S. Census Bureau, rely on proxy reporting. Specifically, the CPS accepts reports from any adult member of selected households, and these individuals are asked to describe many characteristics of all other household members. The accumulated literature on proxy reports suggests that although they may sometimes be quite accurate and comparable to self-reports, this is not always the case. Therefore, there appears to be value in understanding the actual accuracy of proxy reports, how to maximize their accuracy, and when to rely instead only onself reports. 3

4 10:50-12:00PM Coding Open Responses Arthur Lupia University of Michigan (via video) Dean Kotchka - Ascribe Although survey researchers have been asking open-ended questions for decades and coding the obtained answers for later analysis, the procedures used for such coding have rarely if ever been informed by the large and multifaceted literature across the social sciences on best practices for content analysis of open-ended text. For example, it has not been routine for survey response coding to be subjected to assessments of reliability in light of the controversy in the content analysis literature about the merits of various different methods for doing such assessment. Furthermore, rarely have survey researchers released detailed sets of instructions used to guide their coders. With the rise in natural language processing by computers, the notion that the work of human coders may be better done by computers is gaining traction. These are just some of the potential opportunities for improvement in coding practices. 12:00-1:00PM 1:00-2:00PM Lunch Break What HLT Can Do For You (and vice versa) Mark Liberman University of Pennsylvania Some now-standard kinds of automatic text analysis, such as "document classification", "entity and relation tagging", and "sentiment analysis", are closely related to what social science researchers often do with transcripts of open-ended survey responses. Furthermore, automatic speech recognition can produce text that feeds effectively into analysis of this sort, often with relatively little degradation of performance. So in principle, "Human Language Technology" promises to give survey researchers faster results at lower costs. In addition, new sorts of insights may be available from analysis of the statistical patterns of word and concept associations in collections of survey transcripts, or from analysis of the many properties of spoken responses that are left out of the word sequences that make up such transcripts. And the growing scale and scope of networked social media raises the hope that "virtual surveys" can be done by appropriatelyweighted analysis of the data available from such sources. But for the moment, satisfactory results will generally require the involvement of experts in natural language processing and speech technology, to select, adapt and apply the nowstandard techniques, or to develop new ones. Luckily, there is an easy way to entice speech and language researchers into collaborations in this area. In this presentation, I'll give a basic taxonomy of relevant HLT methods, characterize their current level of performance, and describes the conditions needed for them to work well. I'll also describe the process that has led to steady technical progress in this field over the past 25 years, and explain how survey researchers can harness this process and direct it towards the goals that matter to them. 4

5 2:00-3:00PM Address-based and List-based Sampling Colm O Muircheartaigh University of Chicago In recent years, the process by which samples are drawn for face-to-face and mail surveys has increasingly shifted to address-based sampling via lists of addresses available in the USPS Delivery Sequence File, which is thought to be a nearuniversal and frequently updated frame of households. The efficiency offered by this sampling approach is appealing, but the specific procedures used to draw samples and to update listings in the field are not yet standardized across organizations. 3:00-3:15PM Coffee Break 3:15-4:30PM Interviewer Deviations From Scripts Nora-Cate Schaeffer University of Wisconsin-Madison (via video) Hector Santa Cruz Stanford University The dominant method in survey interviewing in recent decades has been standardized interviewing using instruments that are compatible with standardization. Standardization aims to make interviewers interchangeable to reduce the interviewer component of variance and thus increase the reliability of measurement. Practices of standardization have been confronted by at several challenges over the years. One is a concern with validity and comprehension and an interest by researchers in providing resources to interviewers to improve comprehension by respondents. A second is the development of interviewing methods in which data are requested and recorded in more variable forms that are less compatible with standardization, such as event history calendars. A third is evidence from the study of recordings that standardization is sometimes difficult for interviewers to maintain. Evaluating standardization and these new developments in interviewing is complicated by the difficulties of designing experiments that allow for adequate tests of the effects on both interviewer variance and validity or accuracy. 5

6 4:30-5:30PM Panel Attrition Randall Olsen Ohio State University Some of the most important survey studies are long-term panels, including the National Longitudinal Surveys and the Panel Study of Income Dynamics. Part of their value comes from the tracking of the same individuals over very long time periods. But the longer the tracking period gets, the more sample attrition occurs, and the less representative of the population the sample may become. Although some literature suggests that panel attribution has remarkably little impact on sample representativeness, other literature does not support this conclusion. Furthermore, efforts to prevent attrition are not standardized across studies and may not yet be optimized. 6

7 October 4 th Agenda 7:30-8:30AM Continental breakfast 8:30-9:25AM Improving Question Design to Maximize Reliability and Validity Jon Krosnick Stanford University During the last 100 years, the practice of survey questionnaire design has resembled stereotypes of the Wild West: different investigators write different sorts of questionnaires with no apparent set of core principles or evidence apparently identifying best practices for maximizing reliability and validity. Yet over these years, a growing body of research, largely unrecognized by most contemporary questionnaire designers, provides guidance in making such decisions. And new such work is being published all the time. Furthermore, in many situations, researchers choose to ignore this evidence by continuing to ask old, suboptimal questions in order to permit tracking trends over time. Greater uniformity in the practice of questionnaire design and a willingness to replace suboptimal measures in new studies may be desirable. 9:25-10:20AM Probability vs. Non-probability Methods Gary Langer Langer Research Associates Most survey researchers (including the AAPOR Task Force on Online Panels) acknowledge that probability sampling is the best way to collect the most accurate measurements describing a population of interest. However, the cost and time savings afforded by collecting data from convenience samples of volunteers has attracted many researchers to non-probability samples. Various sorts of steps have been taken to enhance the resemblance of such samples to the populations of interest, including propensity score matching, quotas, weighting, aggregation of individuals from multiple recruitment sources, and river sampling of people carrying out other activities. Understanding the effectiveness of these methods seems worthwhile for all of survey research. 10:20-10:30AM Coffee Break 7

8 10:30-11:25AM Building Household Rosters Sensibly Kathy Ashenfelter U.S. Census Bureau The vast majority of representative sample surveys begin by interviewers eliciting rosters of all household members, so a random selection can be made among them. But the rules used to determine who should be considered a household member and who should not are remarkably unstandardized across studies. For example, the U.S. Census Bureau employs different rules in different studies, and even within a single studies, different instructions to interviewers and respondents sometimes contradict one another. Thus, this is an arena in which conceptual and operational work can help survey researchers to optimize a procedure that is used across a large number of studies 11:25-12:20PM Incentives Eleanor Singer University of Michigan This presentation summarizes what we know about the effect of incentives on various outcomes--response rates in different kinds of surveys, sample composition, response quality, and response distributions. The findings are based, whenever possible, on randomized experiments using large samples and, where possible, replicated. Since most of these experiments have been designed to improve response rates, the presentation suggests how one might think about using incentives to achieve different goals, primarily to reduce nonresponse bias. The presentation concludes with a few general "best practices" and a longer list of recommendations for needed research. 12:20-1:20PM Lunch Break 1:20-2:15PM Perception of Visual Displays and Survey Navigation Stephen Kosslyn Stanford University (via video) Surveys often make use of show cards and other visual aids for respondents yet little work has been done to bridge the understanding of visual perception developed by cognitive psychologists and best-practices in the visual design of surveys. Furthermore, despite the development of a significant empirical literature on the effects of survey design and the visual navigation of survey instruments, very little has been done in this area to bridge work from psychology on visual perception and navigation to survey methodology. The time may be ripe for interdisciplinary efforts to join survey methodologists and psychologists to conduct research aimed at developing a set of best-practices for the visual presentation of surveys and survey tools to maximize their effectiveness with respondents. 8

9 2:15-3:10PM Data Collection Mode Roger Tourangeau Westat For several decades, beginning in the 1970s, survey data collection was dominated by three modes of data collection face-to-face interviewing, telephone interviewing, and mail surveys. The widespread adoption of computer technology by survey researchers has multiplied the possibilities for conducting surveys. Many surveys continue to rely on face-to-face or telephone interviews but with computerized rather than paper questionnaires. However, advances in telephone technology, including the rapid proliferation of mobile telephones, has had the paradoxical effect of making telephone surveys harder to carry out than before. As a result, mail surveys sent to address-based samples may come to supplant telephone surveys for many researchers. More recently, the Internet has offered the opportunity for faster and perhaps less expensive data collection. The choice of a data collection mode has effects on all the major sources of survey error sampling, coverage, nonresponse, and measurement error. Various features of the different mode account for their effects on survey error. These include 1) the presence of an interviewer, 2) the ability of a computerized instrument to be responsive (versus the near impossibility of responsiveness in a paper questionnaire), 3) the different demands on reading skill and computer literacy imposed by the different modes, 4) other effects on cognitive burden (such as control over the pacing of the questions in self-administered modes); and 5) the reliance on visual or aural channels to convey information. Understanding these issues can help researchers to choose among modes, to tailor the design of data collection in a particular mode, and to decide when and whether to implement multi-mode designs. 3:10-3:20PM Coffee Break 3:20-4:15PM Optimizing Response Rates Michael Brick Westat Evidence to date suggests that even low response rate surveys are remarkably robust with respect to nonresponse bias, but maximizing response rates is still a standard objective of many sample surveys. In recent years, research has generated many insights into effective practices for maximizing response rates. These goals of achieving maximal response rates have obvious benefits, but they may also have some negative consequences. This talk discusses some of the key factors that affect response rates and the potential effects of those techniques. Clearly, the field is not yet unified in approaches that can be efficient and effective. 9

10 4:15-5:10PM Computation of Survey Weights Matthew DeBell Stanford University A standard component of survey research is the computation of weights to poststratify survey samples and to account for intended unequal probabilities of selection due to study design. Until recently, the process of building weights was thought to be as much arts as science, adjusting a particular sample in order to maximize accuracy while minimizing the design effect. However, some largescale infrastructure survey projects do not provide carefully constructed weights for some of their survey datasets. And although the procedures used to compute can be standardized to a large extent (e.g., by the R software package ANESRake), such standardization is not yet common across studies. For example, a public debate erupted recently when it was revealed that the Gallup Organization builds weights to match the demographic characteristics of owners of landline telephones, despite also calling cell phones. One prominent researcher went so far as to say: Survey weights, like sausage and legislation, are designed and best appreciated by those who are placed a respectable distance from their manufacture. 5:10-5:15PM Closing Remarks Jon Krosnick Stanford University 10