Inference and Error in Surveys. Professor Ron Fricker Naval Postgraduate School Monterey, California

Transcription

1 Inference and Error in Surveys Professor Ron Fricker Naval Postgraduate School Monterey, California 1

2 Goals for this Lecture Learn about how and why errors arise in surveys So we can avoid/mitigate them as much as possible Characterize the survey process: (1) from a design perspective, and (2) from a quality perspective Use these perspectives to define the types of error and express them in statistical notation 2

3 Two Types of Survey Inference 2. The survey respondents must have characteristics similar to those of the larger population 1. Answers people give must accurately describe their characteristics 3

4 Sampling for Statistical Inference Population sample Unobserved population statistic inference Sample statistic 4

5 Survey Goal: Elicit Accurate Information Accurate Means Minimizing Total Survey Error (Groves, 2004) Source of Error Coverage Sampling Nonresponse Measurement Definition...from the failure to give any chance of sample selection to some persons in the population....from heterogeneity on the survey measure among persons in the population....from the failure to collect data on all persons in the sample....from inaccuracies in responses recorded on the survey instruments. These arise from: (a) effects of interviewers on the respondents' answers to survey questions; (b) error due to respondents, from the inability to answer questions, lack of requisite effort to obtain the correct answer, or other psychological factors; (c) error due to the weakness in the wording of survey questionnaires; and, (d) error due to effects of the mode of data collection, the use of face to face or telephone communications. 5

6 Survey Error: Two Perspectives Design perspective Thinking about the survey process, as a survey moves from ideas to concrete actions From identifying the ideas of interest to coming up with the specific questions to ask From identifying the population of interest to collecting the data and making necessary adjustments Quality perspective A taxonomy to distinguish survey designs by major sources of error 6

7 Survey Lifecycle from a Design Perspective This side is about what is being measured This side is about who is being measured 7

8 Measurement Example #1 (CES) Construct: How many new jobs were created in the US in the last month? Measurement: How many new jobs were created in your company in the last month? By new jobs, we mean Response: Respondent contacts a human resource person in their firm to obtain the data Edited Response: Subsequent questions asked about the type of jobs created Using this information, combined with the answer to the first measurement question, number of new jobs created in this firm in the last month calculated 8

9 Measurement Example #2 (NCVS) Construct: How many incidents of crime were there in the past year in the US? Measurement: During the last year, did you call the police to report something that happened to you that you thought was a crime? Response: Person thinks back over the past year to come up with a response Edited Response: Subsequent questions are asked to qualify whether the calls were related to a crime Using this information, combined with the answer to the first measurement question, number of crimes (as defined in the survey) that happened to this respondent calculated 9

10 Construct Example #1 (NCVS) Target Population: Individuals aged 12 and over, not on active military service, who reside in noninstitutionalized settings Frame Population: US households enumerated through counties, blocks, listed addresses, listed members of each (selected) household Sample: About 42,000 households 76,000 individuals Respondents: Each person aged 12 or older reports for self Postsurvey Adjustments: Missing data estimated ( imputed ), nonresponse adjustments, etc. 10

11 Construct Example #2 (SOC) Target Population: Nonistitutionalized adults in the continental US (omits Hawaii and Alaska) Frame Population: Continental US adults in a household with a (landline) telephone Sample: 500 adults randomly drawn using random digit dialing (RDD) based on lists of working area codes and exchanges Respondents: One adult randomly selected from the household Postsurvey Adjustments: Results adjusted for units known not to be in the sampling frame (e.g., new households) 11

12 How Design Becomes Process 12

13 Survey Lifecycle from a Quality Perspective Design flow of measurement and representation also useful for thinking about surveys from a quality perspective Mismatches in the transitions from step to step represent possible errors Can also express each step using statistical notation, from which the errors can then be quantified Notation: Greek letters (i.e., µ): Unknown population quantities Capital Roman letters (i.e., Y): Random variables Small Roman letters (i.e., y): Observed values of the random variables 13

14 Survey Lifecycle from a Quality Perspective This side is about individual measurements This side is about aggregate measurements 14

15 Observational Gap Between Constructs and Measures Validity is the extent to which the measure is related to the underlying construct Example: NAEP measures mathematical ability of 4 th graders using arithmetic problems Purview of the field of psychometrics Idea: for each individual the construct measured with error, Y = µ + ε i i i Given this, we could think about testing an individual multiple times Y it = µ i + εit Mathematically, then, we can define validity as ( )( ) ( ) 2 2 E it Yit Y µ i µ / Eit Yit Y Eit ( µ i µ ) 15

16 Measurement Error Measurement error is the observational gap between the ideal measurement and the response obtained For example, the NSDUH question Have you ever, even once, used any form of cocaine? may be underreported due to sensitivity to question Possible response bias Response bias occurs when E Y ( ) t it i Responses are systematically off (low or high) Different from excessive response variance Y 16

17 Processing Error Processing error is the observational gap between the variable used in estimation and that provided by the respondent Example: A presumed outlier that is subsequently adjusted in the analysis was actually correct Example: In an in-person survey, the interviewer misunderstands or mis-records a response Example: In a paper survey, a response to an open-ended question is incorrectly summarized / categorized / coded 17

18 Coverage Error Coverage error is the nonobservational gap between the target population and the sampling frame Applies to both sample and census surveys Mathematically, we can write is as Y Y = U Y Y N ( ) C C U 18

19 Sampling Error Sampling error is the nonobservational gap between the sampling frame and the sample 19

20 Sampling Bias and Variance Sampling bias mainly affected by how probability of selection assigned to sampling frame elements Easily handled by giving equal probability of selection to all elements (simple random sampling or SRS) Sampling variance reduced with larger sample sizes, sometimes with stratification, and by not clustering With SRS, sample means are y s n 1 s = n s i = 1 y si and 1 C C i = 1 Sample variance, S, is a measure of how i = 1 precise our estimates are Y C 1 S s C ( y Y ) 2 = Y i 20

21 Nonresponse Error Nonresponse error is the nonobservational gap between the sample and respondent pool In surveys of people, some just don t respond Nonresponse bias occurs when the statistics computed from the response data differ from those based on the entire sample: m ( ) s yr ys = yr ym ns Cannot observe nonresponse bias in survey data; only for observable sample characteristics High response rates reduce the risk of nonresponse bias 21

22 Adjustment Error Adjustment error are nonobservational errors that result from postsurvey adjustments Example: Use of weighting to adjust for nonresponse In NCVS, response rate for urban areas is 85 percent while for rest of country it s 96 percent Could weight data to counteract, setting w for urban respondents and i = 1/ 0.85 w i = 1/ 0.96 for all others r r Adjusted sample mean is then y w y w = rw i si i i= 1 i= 1 Often an improvement, but not necessarily 22

23 Uses of Survey Data Descriptive How prevalent is an attribute in a population? Think of it in terms of descriptive statistics Analytic How are two attributes associated with oneanother? Think of it in terms of models, such as regression Errors in survey data can affect both types of survey analyses / results 23

24 What We Have Covered Learned about how and why errors arise in surveys Useful to understand so we can design and field surveys in ways that avoid/mitigate the errors as much as possible Characterized the survey process: (1) from a design perspective, and (2) from a quality perspective Used these perspectives to define the types of error and we expressed them using statistical notation 24