Survey Errors and Survey Costs

Transcription

1 Survey Errors and Survey Costs ROBERT M. GROVES The University of Michigan WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION

2 CONTENTS 1. An Introduction To Survey Errors Diverse Perspectives on Survey Research The Themes of Integration: Errors and Costs The Languages of Error Classifications of Error Within Survey Statistics A More Restrictive View, Sampling Statistics Total Survey Error Other Error Terms in Survey Statistics Terminology of Errors in Psychological Measurement Estimating Validity Other Notions of Validity The Language of Errors in Econometrics Debates About Inferential Errors Important Features of Language Differences Summary: The Tyranny of the Measurable Summary and Plan ofthis Book 34 Appendix for Mathematical Presentation of Error Perspectives A Classifications of Error Within Survey Statistics 38 l.b Terminology of Errors in Psychological Measurement 41 l.c The Language of Errors in Econometrics An Introduction To Survey Costs Rationale for a Joint Concern About Costs and Errors 49 xiu

3 xiv CONTENTS 2.2 Use of Cost and Error Models in Sample Design Criticisms of Cost-Error Modeling to Guide Survey Decisions Nonlinear Cost Models Often Apply To Practical Survey Administration Survey Cost Models are Inherently Discontinuous Cost Models Often Have Stochastic Features Domains of Applicability of Cost Models Must Be Specified Simulation Studies Might Be Best Suited to Design Decisions Is Time Money? Summary: Cost Models and Survey Errors Costs And Errors Of Covering The Population Definitions of Populations Relevant to the Survey Coverage Error in Descriptive Statistics An Approach to Coverage Error in Analytic Statistics Inferential Populations in Analytic Statistics Treatment of Inferential Populations in Survey Analysis A Link Between Coverage Error and Analytic Modeling ' Components of Coverage Error Coverage Problems with the Target Population of Households Coverage of Housing Units Coverage Errors for Persons Within Housing Units Telephone Number Frames for the Household Population 116

4 CONTENTS XV 3.6 Measurement of and Adjustments for Noncoverage Error Noncoverage Error Estimation Multiplicity Sampling and Measurement Multiple Frame Methods The Half-Open Interval Use of Post Survey Adjustments for Coverage Error Survey Cost Issues Involving Coverage Error Summary Nonresponse in Sample Surveys Nonresponse Rates Components of Unit Nonresponse Nonobservation of Sample Units Classification of Outcomes Response Rate Calculation Temporal Change in Response Rates Ongoing Surveys of Government Survey Organizations Ongoing Surveys of Academic Survey Organizations t Surveys Conducted by Commercial Agencies Item Missing Data Statistical Treatment of Nonresponse. in Surveys Properties of Components of Nonresponse Error Statistical Models for Nonresponse A Sequential Design Option Incorporating Nonresponse Concerns Summary 182

5 xvi CONTENTS 5. Probing The Causes Of Nonresponse And Efforts To Reduce Nonresponse Empirical Correlates of Survey Participation Patterns of Being At Home Demographic Characteristics of Refusers Efforts by Survey Methodologists to Reduce Refusals Contact with Sample Households Before the Survey Request Descriptions of the Survey at Time of Introduction Attempts to Increase the Benefits of Cooperation Discussion of Attempts to Reduce Nonresponse from Refusals Sociological Concepts Relevant to Survey Nonresponse Psychological Attributes of Nonrespondents The Literature on Altruism and Compliance The Psychology of Persuasion Summary Costs And Errors Arising From Sampling Introduction The Nature of Sampling Error Measuring Sampling Error Nonprobability Samples Comparing Probability and Nonprobability Samples?'* Four Effects of the Design on Sampling Error Stratification Probabilities of Selection Clustering Sample Size Combinations of Sample Design Features 268

6 CONTENTS XV The Effect of Nonsampling Errors on Sampling Error Estimates Measuring Sampling Errors on Sample Means and Proportions from Complex Samples The Debate on Reflecting the Sample Design in Estimating Analytic Statistics Reflecting the Sample Design When Estimating Complex Statistics Summary Empirical Estimation Of Survey Measurement Error A First Examination of Observational Errors Versus Errors of Nonobservation Laboratory Experiments Resembling the Survey Interview' Measures External to the Survey Randomized Assignment of Measurement Procedures to Sample Persons Split Sample Experiments Concerning Measurement Error Interpenetration for Estimation of Variable Measurement Error Practical Problems with Randomization as a Procedure for Estimating.Measurement Error Repeated Measurements of the Same Persons Reinterviews To Estimate Measurement Error Multiple Indicators of the Same Characteristics Administered in the Same Questionnaire Measuring Characteristics of the Interview Itself Direct Measures of Cognitive Steps in Response Formation Evaluating Repeated Measurements as a Procedure for Estimating Measurement Error 344

7 xviii CONTENTS 7.6 Summary of Individual Techniques of Measurement Error Estimation Combinations of Design Features Interpenetration for Measurement Bias Estimation and Interpenetration for Variable Interviewer Effect Estimation Interpenetration of Interviewer Assignments and Reinterviews Interpenetration of Interviewer Assignments and Multiple Indicators Summary The Interviewer as a Source of Survey Measurement Error Alternative Views on the Role of the Observer The Role of the Survey Interviewer Designs for Measuring Interviewer Variance Interviewer Effects in Personal Interview Surveys Interviewer Effects in Centralized Telephone Surveys Magnitudes and Precision of Estimated Interviewer Effects Stability of Interviewer Effect Estimates Explaining the Magnitude of Interviewer Effects The Influence of Question Form on Interviewer Effects ' Effects of Interviewer Behavior Guidelines on Interviewer Variability Effect of Computer-Assisted Telephone Interviewing (CATI) on Interviewer Variability Interviewer Characteristics and Interviewer Effects Respondent Characteristics and Interviewer Effects 379

8 CONTENTS xix 8.7 Summary of Research on Interviewer Variance Measurement of Interviewer Compliance With Training Guidelines Experiments in Manipulating Interviewer Behavior Effects of Interviewer Training Experiments with Interviewer Feedback to Respondents Social Psychological and Sociological Explanations for Response Error Associated with Interviewers Effects of Interviewer Expectations Demographic Characteristics of the Interviewer Summary The Respondent as a Source of Measurement Error Components of Response Formation Process The Encoding Process and the Absence of Knowledge Relevant to the Survey Question Respondent Rule Effects Nonattitudes Comprehension of the Survey Question Retrieval of Information from Memory Length of Recall Period Saliency of Events to Be Recalled Task Difficulty A Respondent Attention or Motivation Judgment of Appropriate Answer Communication of Response Social Desirability Effects Acquiescence, Yea-Saying, Nay-Saying, Response Tendencies 439

9 xx CONTENTS 9.7 Sociological and Demographic Correlates of Respondent Error Summary Measurement Errors Associated With the Questionnaire Properties of Words in Questions Properties of Question Structure Length of the Question Open Versus Closed Questions Number and Order of Response Categories in Fixed Alternative Questions Explicit "Don't Know" Response Categories Question Structure in Measurement of Sensitive Topics Question Structure and Recall of Past Events Properties of Question Order Context Effects on Responses Position of the Question in the Questionnaire Conclusions About Measurement Error Related to the Questionnaire Estimates of Measurement Error from Multiple Indicator Models Cost Models Associated With Measurement Error Through the Questionnaire Summary Response Effects of the Mode of Data Collection Two Very Different Questions About Mode of Data Collection Properties of Media of Communication 507

10 CONTENTS xx i 11.3 Applying Theories of Mediated Communication to Survey Research Findings from the Telephone Survey Literature Interaction Effects of Mode and Other Design Characteristics on Responses Social Desirability Effects on the Telephone Cognitive Demands of the Survey Question and Mode Mode Effects and Interviewer Effects Survey Costs and the Mode of Data Collection A Cost Model for a Centralized Telephone Facility Empirical Data on Cost of Data Collection in Different Modes Cost and Error Modeling with Telephone and Face to Face Surveys Dual Frame Survey Design Administrative Structures for Dual Frame, Mixed Mode Designs An Example of a Dual Frame, Mixed Mode Design Simulation Summary 551 References 553 Index 581