in the 21 st Century JPSM Distinguished Lecture April 11, 2008

Transcription

1 Survey Design alamode Survey Research in the 21 st Century Colm O Muircheartaigh JPSM Distinguished Lecture April 11, 2008

2 Outline Historical background Development of the scientific survey The standard d survey 1950s 1960s Disruption and transformation 1970s 1990s Challenges Conclusion

3 I Historical background Origins

4 Origins 1 Official Statistics Social Policy Social Research Market and Opinion Research Kiaer 1895 Booth 1889 Gale 1895 Hull House 1892 Parlin 1915 Monography Population Statistics on a Small Scale Community Description Commercial Information

5 Origins 2 Official Statistics Social Policy Social Research Market and Opinion Research Sampling Bowley 1906 Neyman 1934 Standardized tests Psychological Corporation 1932 Facts Bowley 1915 Attitudes Bogardus 1925 Thurstone 1928, Likert 1932 Scientific Sample Survey Scientific Opinion Polls 1930s Advertising Media Consumer Social GOVERNMENT ACADEMIC BUSINESS Statisticians Sociologists Psychologists

6 II Development of the Scientific Survey The statistical perspective The basic design Model of the survey The task The interviewer

7 History 2 Development Neyman (1934) Superiority of probability sampling Theory for unequal cluster sampling Gallup and others Scientific samples for opinion polls Hansen Hurwitz Madow 1940s PPS at higher stages Adequate representation of important units Leads to identification of certainty PSUs Equal workloads at final stage (HUs) Efficiency of field allocation and estimators 1950s: national master samples ISR, NORC, et al.

8 Statistical Perspective the Responses Survey statistics and the description of populations Bowley (1905): inference for SRS Neyman (1934): inference for clustered samples Mahalanobis (1946): replicated variance estimates Yates (1946, 1949): ANOVA for components of survey variance Hansen, Hurwitz, Madow (1954): formulas for variances for complex samples Hansen (1961), Kish (1962), Fellegi (1964): correlated response variances

9

10 Sudman-Bradburn Model INTERVIEWER RESPONDENT TASK RESPONSES

11 The Task: Art or Science? Seen as art (craft) Common sense Clear Simple Not double-barreled Assume as little as possible Use filters

12

13 Role of the Interviewer EXPERT interviewer vs QUESTIONNAIRE interviewer [1890s s?] The neutral instrument 1940s Control/selection/limiting quotas [1940s, 1950s] Interviewer as motivator [1920s/1970s]

14 III The Standard Survey 1950s 1970s Government and social research Market research Sampling frames Modes Estimation Mode/frame specificity

15 Government and Social Research Survey organizations Government statistical offices f/f probability samples / Census frames f/f probability samples / administrative or electoral lists Some self-completion, especially Census and establishments Not for profit; social research; developing countries Some countries, list-based f/f probability samples; frame construction at UAU Some self-completion Master samples (Semi-)permanent field staff High response rate, good coverage

16 Market Research f/f quota samples; shorter field period; response rates not defined, but refusal rates fairly low; coverage good among target population Justifiable on trapping principle for wildlife samples? Mall tests; quota based Some self-completion eg diaries, but f/f recruitment Telephone directory or customer frame samples; short field period; moderate response rates Cheap, suitable for market research Rudimentary phone

17 Self-completion surveys Mail surveys still widely used Supplements to f/f Sensitive e questions Detailed information eg diaries for CE / HBS Audience measurement (radio, television) Where record consultation required (leave-behind qu res) Establishment surveys Census

18 Sampling frames Mode/frame specific

19 The Basic National Demographic Design Multi-stage Costs Feasibility Some self-representing PSUs Stratified Incorporating knowledge of population and structure USA Decennial update of frame Master sample Elsewhere Register-based Scandinavia, China, UK, Deteriorating registers: UK, postal address frame Periodic redesign

20 Estimation Neyman (1934) Estimate Standard error Confidence interval

21 IV Transformation 1970s 1990s Statistics: challenge to the Neyman paradigm Task: challenge to the conceptualization of questionnaire Interviewer: challenge to the nature of the interview Respondent: re-interpretation of respondent role Technology Modes: new, old, and in-between Sampling: new frames, new possibilities

22 Statistics model-based inference Sampling Challenge to Neyman inference in survey statistics Godambe (1955), Ericson (1968), Royall (), Smith (1975) Model-based inference Nonresponse Undermined the purity of design-based inference Model-based adjustments Rubin, Little Noncoverage Included in missingness

23 The Task Revisited CASM [Cognitive Approaches to Survey Methods] Theory-informed approach to questionnaire effects Re-interpretation of interview content Question wording: equivalent stimulus not equivalent wording and presentation Language of qu res Grade-level or community-relevant Biomeasures Interviewer observations

24 Role of the Interviewer Revisited Interaction coding [Cannell 1975] Interviewer style [Dijkstra and van der Zouwen 1980s] CASM Changing the understanding of the interviewerrespondent interaction and the task Empowerment/local control Suchman and Jordan 1990 Schaeffer 1991 For the two perspectives: Fowler and Mangione (1990) Maynard, Houtkoop, Schaeffer, van der Zouwen (2002)

25 Respondent Diversity Visited Some recognition of respondent relevance Challenge to the monolithic model Implications of Grice s maxims eg languages and translation WFS Previously: one master questionnaire developed in the source language, then carefully translated into other languages for use Implications of wording Different subpopulations

26 New Modes Telephone Alexander Graham Bell 1876; full direct distance dialing 1960s Consolidated directories and RDD Fast, cheap, low cost of entry (compared to f/f) Internet New, exciting Excellent delivery of complex stimuli Opt-in polls / access panels Fast, cheap, big samples, modish

27 Technology CASIC [Computer-assisted t survey information collection] Adaptable to multiple modes Flexible and easily controlled delivery of tailored stimulus Computer-assisted sampling Fast Portable in principle New challenges in application, management, and integration

28 V The Challenges Proximate criticisms New forms of social research Multiple modes and response comparability across modes Alternative statistical inferences

29 Proximate criticisms Phone Noncoverage due to cell-phone only Low response rates Internet alternative Fast, cheap Large sample sizes Flexible stimulus delivery Flexible respondent demand (timing, location) cf. mail surveys Face-to-face Too expensive, too slow

30 New forms of social research? Opt-in polls / access panels Poll aggregation/meta-analysis UK poll of polls Realclearpolitics.com Pollster.com Existing databases Data mining Geo-statistics Social network analysis Geocoding built into other instruments Robocalls

31 Multiple modes /multi-mode surveys Hochstim (1967) Sequences of modes ACS Master frame Graduated effort; cost gradient Methodological comparisons of modes Narrow Same cases with different modes (experimental designs) Broad Overall survey operation with different modes Coverage, nonresponse, measurement quality, feasibility, cost Survey profiles Perhaps the wrong question

32 Multimode Surveys Pi Primary + supplementary Sequences of modes Battery of modes Examples ACS sequence (descendant of Hochstim) One for some, another for others (CES) One for recruitment, another for survey (IPFF) One for data collection, one for reminders One for one wave of a panel, another for other waves (CPS) One for some data, another for the rest (leave-behind SAQ; ACAS component) HRS;NSHAP

33 Components of Statistical Inference Sampling inference Design-based Model-based Model-assisted Inference for non-response Weighting Imputation Inference for noncoverage?

34 VI Conclusion Diversity Multiple modes Inference

35 Matching diversity with diversity The world/population as a multi-dimensional lattice Multiple frames Multiple characteristics Multiple tasks

36 Respondent Diversity Revisited New paradigm Cross-national surveys ESS, Eurostat surveys Cross-cultural surveys NIS, ACS eg Language Not a master questionnaire Qu res to be developed simultaneously for multiple contexts Houston, TX, the US city of the future? 75% of over-65s Anglo 75% of under 30s not Anglo

37 Dimensions of Variation Nature and quality of the frames Address; phone number; internet Propensity to respond To mail; phone; to internet; to face to face interview Population diversity Languages, cognition, robustness to burden Some mode-specific considerations, e.g. Coverage for internet Population density for f/f

38 Some Cells of the Multidimensional Lattice Not on the street-style address frame, no phone, no internet access, low propensity to respond to face to face interviewing Available only by field listing and face to face interviewing, poor prospect for survey research On street-style address frame, listed phone number, internet access, high propensity to respond to all approaches Available through multiple frames; data available through multiple modes, the ideal prospect

39 Multiple Modes, Multiple Designs Lattice designs Streams of information Dispositions Projections Dynamic adjustments Switching among modes Switching among burdens

40 Rethink the Survey Operation I Focus on objective, not on variation in methodology Match the heterogeneity of the population with the diversity of our approaches Typology of the data/respondent array Technology can enable this nimbleness of response Separate respondent capture from data capture Interviewers Conflicting skills necessary Solve/manage the transition problem Modes IPFF NLTCS Utilize diverse sources of information

41 Rethink the Survey Operation II Utilize diverse sources of information Smart cards, administrative data (public and private) Develop biomeasure capacity But change in nature and impact of interviewer? NRFU principles? More of easier cases, or Some of the more difficult cases Modeling vs design-based inference Need to learn both Need to embrace both

42 Inference Do not be intimidated id t d by algebra Survey statisticians need to understand more than just Neyman The tyranny of language eg ignorability Don t accept and use terms that are misleading Remediable? Repairable? Need to address the components of the inference problem Explicit recognition of strengths and weaknesses of appproaches 30% RR vs Medical Centers Opt-in polls vs low RR RDD Don t deny successes of other methodologies

43 Legacy of past survey research: springboard or trap?