Tim Johnson Survey Research Laboratory University of Illinois at Chicago September 2011

Transcription

1 Tim Johnson Survey Research Laboratory University of Illinois at Chicago September

2 Characteristics of Surveys Collection of standardized data Primarily quantitative Data are consciously provided by subjects about themselves Representative sampling Statistical analysis 2

3 Uses of Surveys Description Needs assessments Social monitoring Prediction Evaluation research Market & political research Hypothesis/model testing 3

4 Advantages of Surveys Uniqueness gather information not available from other sources. Probability Sampling unbiased representation of population of interest. Standardization of Measurement same information collected from every person Can examine multiple topics Current/up-to-date information 4

5 Advantages of Surveys #2 Examine phenomena not directly observable - attitudes and opinions probable reactions future plans/expectations past behaviors/experiences sensitive behaviors Analysis needs use surveys to compliment existing data from other sources Better accuracy than a census or records in many cases 5

6 Disadvantages of Surveys Cost Expertise/experience required Administration Accessibility of respondents children mentally incompetent residentially isolated institutionalized popuation 6

7 Disadvantages of Surveys #2 Public resistance Over-surveyed populations Privacy concerns Technological barriers Discrepancies between reported and actual behaviors Memory errors Misrepresentation Hypothetical unreliability Proxy reports 7

8 A Brief History Surveys the single most important invention of the social sciences during the 20th Century...born out of the needs of modern state bureaucracies surveys and polls are also an important process in democratic political systems 8

9 Ancient Ancestors Censuses conducted at least 2000 years ago by Roman Empire General Census of England conducted by William the Conqueror Also in Florence and Venice during the Renaissance During English Industrial Revolution, surveys of the conditions among the poor were conducted (cf., Charles Booth) 9

10 10

11 Modern Surveys Evolved from Psychologists and market researchers, who contributed the techniques of asking about preferences using standardized questionnaires. Journalists, who contributed the idea that people could be questioned about their candidate preferences. Sociologists, who contributed the idea that social conditions could be measured. 11

12 By the 1930s All that was missing was the concept of probability sampling. The infamous Literary Digest Poll (1936). 12

13 During the 1940s Kinsey Report demonstrated that sensitive topics could be studied Pollsters failed to predict winner of 1948 Presidential Election let to strengthened sampling procedures 13

14 14

15 In the 1960s Development of high speed computers Increased government use of surveys 15

16 In the 1970s Declining survey response rates in many countries Development of telephone surveys and random-digit-dialing (RDD) sampling methods 16

17 In the 1980s Development of computerized interviewing systems Application of cognitive methods to improve survey measurement 17

18 Development of audio computer-assistedpersonal-interviewing (ACASI) methods for collecting sensitive information In the 1990s 18

19 Since 2000 Proliferation of mobile phones and reduction in landline phones has created serious accessibility and coverage problems Web is now probably the most common mode of survey data collection 19

20

21

22 Basic Survey Designs Cross-Sectional Surveys: Data are collected at one point in time from a sample selected to represent a larger population. Longitudinal Surveys: Trend, Cohort, and Panel 22

23 Types of Longitudinal Surveys Trend surveys of sample population at different points in time Cohort study of same population each time data are collected, although samples studied may be different Panel collection of data at various time points with the same sample of respondents. 23

24 What Drives Survey Design Decisions? A set of clearly-defined study goals and objectives A clearly-defined study population Cost considerations Time considerations Quality considerations 24

25 Modes of Survey Administration Personal (face-to-face) Telephone Mail Web Combination of methods 25

26 Modes of Survey Administration Personal (face-to-face) Telephone Interviewer-assisted Mail Web Combination of methods 26

27 Modes of Survey Administration Personal (face-to-face) Telephone Mail Web Interviewer-assisted Self-administered Combination of methods 27

28 Total Survey Data Collected, by Mode, by U.S. Academic Centers 28

29 Trends in Survey Data Collection, by Mode, of U.S. Academic Centers 90,0% 80,0% 70,0% Survey Mode, by Year: Regression Lines 60,0% 50,0% 40,0% 30,0% Web Phone In Person Mail 20,0% 10,0% 0,0%

30 Paper vs. Computer Administration PAPI: paper-and-pencil interviewing CAI: computer-assisted interviewing CATI: computer-assisted telephone interviewing CAPI: computer-assisted personal interviewing CASI: computer-assisted self-interview ACASI: audio computer-assisted self-interview IVR: interactive voice recognition (telephone) 30

31 Advantages of Computer Administration Operational issues Cost comparisons Time to complete Reduction in interviewer errors Branching, insertion of data, instant editing Data available faster after collection 31

32 How do you decide on the mode of data collection? Population + Characteristics of the sample + Types of questions + Question topic + Response rate + Cost + Time + Quality 32

33 Sources of Error in Survey Research 33

34 5 Generally Recognized Sources of Error 1. Sampling error 2. Coverage error 3. Nonresponse error 4. Measurement error 5. Processing error 34

35 Sampling Error Degree of confidence we can place in the survey s findings. Is a function of: 1. How well the study is designed 2. The sample size and design 35

36 Some Basic Sample Designs Probability Designs Simple random samples Systematic samples Stratified samples Cluster samples Nonprobability Designs Convenience designs Quota samples Snowball samples 36

37 Sample Designs for Probability Samples Simple random samples Systematic samples Stratified samples Cluster samples 37

38 Fraction Example of Central Limits Theorem Samples=10000, Population=Normal, N=100 Lines drawn at -2sd -1sd mean +1sd +2sd

39 If a probability sample, sampling error is a function of sample size and design: Larger sample sizes produce more precise population estimates (i.e., estimates with smaller confidence intervals. Cluster designs may produce larger confidence intervals. Stratified designs may produce smaller confidence intervals. 39

40 95% Confidence Intervals for Typical Random Samples: Sample Size Confidence Interval 400 +/- 5.0 percentage points 625 +/- 4.0 percentage points /- 3.0 percentage points 40

41 Coverage Error Errors in the construction of sample frame Failure to represent the population of interest Also known as selection bias 41

42 Types of Selection Bias 1. Defects in sample frame non-telephone households missing telephone exchanges poor mapping of geographic areas sampling homeless in shelters only 42

43 Types of Selection Bias 2. Defects in selection of sample units nonrandom respondent selection procedures misidentification of eligibility use of multiple sample frames 43

44 Relation between target population and sample frame Target Population Sampling Frame Nonresponse Sampled Population Ineligible Borrowed from Gelman & Nolan (2002) Teaching Statistics. 44

45 Nonresponse Error Selected & eligible respondents not included in sample. Do those excluded from the sample differ from those who are included? 45

46 Types of Nonresponse Error 1. Unit nonresponse 2. Item nonresponse 46

47 Sources of Nonresponse Error 1. Refusals to participate 2. Lack of follow-up with nonrespondents 3. Failure to locate respondents 4. Questionnaire design effects 47

48 From Groves, R.M., Fowler, Jr., F.J., Couper, M.P., Lepkowski, J.M., Singer, E., & Tourangeau, R. (2004). Survey Methodology (p. 185). New York: Wiley.

49 From Groves, R.M., Fowler, Jr., F.J., Couper, M.P., Lepkowski, J.M., Singer, E., & Tourangeau, R. (2004). Survey Methodology (p. 187). New York: Wiley.

50 Factors Associated with Nonresponse Error Social Environment: urbanity and population density Interviewer Characteristics: age, gender, self-confidence Respondent Characteristics: social isolation/alienation, privacy concerns, social participation 50

51 Common Indicators of Nonresponse Error 1. Response Rate 2. Cooperation Rate 3. Refusal Rate 51

52 Response Rate Most important indicator of nonresponse error: RR = # Completed Interviews Known eligible + estimated eligible 52

53 Frequency Web Survey Response Rates: Mean =42.36 Std. Dev. = N =116 Web Survey Response Rates

54 Cooperation Rate Commonly confused with response rate: CR = # Completed Interviews # Completed Interviews + # Refusals 54

55 Refusal Rate Also important and should be reported: Ref R = # Refusals Known eligible + estimated eligible 55

56

57 Example of no relationship between nonresponse and measurement error From Merkle, D.M., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In Groves, R.M., Dillman, D.A., Eltinge, J.L., & Little, R.J.A. (Eds.), Survey nonresponse (pp ). New York: Wiley.

58 Measurement Error The difference between reported and true measure Inaccurate answers to survey questions. 58

59 Florida Butterfly Ballot

60

61 Sources of Measurement Error 1. Mode of data collection 2. Survey questions 3. Interviewer 4. Respondent 61

62 Measurement Errors Associated with Mode of Data Collection Mode = self vs. interviewer-administered 1. Degree of anonymity 2. Degree of comfort with computer technology 62

63 Measurement Errors Associated with Survey Question Question sensitivity Question salience Question context Question wording/reading level Question complexity/burden Response option ordering # of response options 63

64 Example of Wording Experiment How would you rate the job Wendell Ford is doing as United States Senator from Kentucky excellent, good, (fair/not so good), or poor? Fair Response Version Not so good Excellent 15.1% 14.9% Good Fair/not so good Poor Don t know N (325) (355) 64

65 Example of Question Context Experiment Do you approve or disapprove of the way Wallace Wilkinson is handling his job as Governor? Question Placement Front Back Approve 43.8% 52.4% Disapprove Don t know N (322) (332) 65

66 Question Wording Effects Question meaning Question balance Loaded questions Double questions Inappropriate vocabulary Missing alternatives 66

67 Measurement Errors Associated with Interviewers Experience Quality of training Personal characteristics Social distance from respondent Respondent familiarity Data falsification 67

68 Self-Reported Past 18-Month Drug Use by Interviewer-Respondent Social Distance Index Source: Johnson, Fendrich, Shaligram, Garcy & Gillespie. (2000) An evaluation of the effects of interviewer characteristics in an RDD telephone survey of drug use. Journal of Drug Issues, 30 (2000),

69 Measurement Errors Associated with Respondent Cognitive Model of Survey Response 1. Interpretation 2. Memory retrieval 3. Judgment formation 4. Response editing 69

70 Processing Errors After data collection...errors associated with: the production of data files, and/or the analysis of data files 70

71 Types of Processing Errors 1. Errors in coding 2. Errors in editing 3. Errors in imputation 4. Errors in weighting 5. Errors in construction of analytic variables 71

72 Procedures to Prevent & Detect Coding/Editing Errors range edits consistency checks between variables outlier detection use of subject matter experts training and supervision of coders re-entry of data inter-coder reliability checks 72

73 Single Best Procedure to Prevent Weighting/Imputation/Variable Construction Errors: Careful rechecking of work 73

74 Example of Processing Error from the U.S. National Household Survey on Drug Abuse 74

75 What happens when you do not weight correctly?

76 In general... Well-documented surveys tend to have fewer processing errors. 76

77 Summary When (1) planning or (2) reporting or (3) evaluating survey research, it is important to consider all five potential sources of error. 77

78 Please contact me at: 78