Time-Sharing Experiments for the Social Sciences. Impact of response scale direction on survey responses to factual/behavioral questions

Transcription

1 Impact of response scale direction on survey responses to factual/behavioral questions Journal: Time-Sharing Experiments for the Social Sciences Manuscript ID: TESS-0.R Manuscript Type: Original Article Specialty Area: Methods/Measurement

2 Page of Time-Sharing Experiments for the Social Sciences Impact of response scale direction on survey responses to factual/behavioral questions Revised proposal for the TESS's Special Competition for Young Investigators Motivation Survey literature has demonstrated that many design features of response scales (e.g., number of scale points, numeric and verbal labels) affect how survey respondents process the scale and use these features to construct their responses. One design feature that is understudied is the direction of a response scale, especially for behavioral questions. A response scale could run from all of the time to never. It could also run from never to all of the time. Holding other features of a response scale constant, an important question is whether or not the direction of the scale affects survey responses, and, if yes, in what way. At least three theories predict an impact of scale direction on survey responses. According to the notion of satisficing (Krosnick, ), respondents process each scale point sequentially and stop processing once they reach an acceptable scale point. Satisficing respondents should, therefore, be more likely to use the starting point(s) of a response scale. A second theory involves anchoring and adjustment (Tversky & Kahneman, ), which suggests that people make numerical estimates by making adjustments to an initial value (i.e., the anchor). The adjustment process tends to be incomplete, resulting in final answers biasing towards the anchor point. The near means related heuristic (Tourangeau, Couper, & Conrad, 00; 00) suggests that the spatial proximity of the survey items and the starting point of a response scale leads to a higher endorsement of this side of a scale in self-administered modes. All three theories predict a tendency for responses to bias towards the starting point of a scale. However, the empirical evidence is mixed. Some researchers found an effect of scale direction on answers (e.g., Belson, ; Carp, ; Friedman, Herskovitz, & Pollack, ; Keusch, 0; Sheluga,

3 Page of Kalton, Collins, & Brook, ; Toepoel, Das, & van Soest, 00; Yan, 00) but others did not (e.g., Christian, Parsons, & Dillman, 00; Dickson & Albaum, ; Ramstedt & Krebs, 00; Weng & Cheng, 000). Thus, more empirical evidence is needed to guide survey researchers design decisions when designing response scales. The existing research studying the impact of scale direction suffers from several weaknesses. First and foremost, the majority of the literature on scale direction has focused on attitudinal questions. Only Carp () included factual questions in her study, but she didn t find an order effect. To advance research in this topic, I propose an experiment examining the impact of scale presentation order on answers to factual/behavioral questions. In particular, I will manipulate the presentation order of a commonly-used frequency scale and study whether or not answers will be different as a function of the presentation order. The second weakness of the literature on scale direction lies in lack of attention to other scale features that could moderate the impact of scale order. The larger study size for the Special Competition allows me to vary two additional features of the response scales (alignment of the scale and vagueness of verbal scale labels). Examining whether or not these factors moderate the scale order effects will carry the research on frequency scales further. One feature pertains to the alignment of the scale. Literature offers inconclusive evidence on the possible moderating impact of the scale alignment on the scale order effect. For instance, Christian et al. (00) showed that horizontal scales are less prone to scale presentation order effect than vertical scales. However, Toepoel et al. (00) found a scale order effect for vertical rating scales and Stapleton (0) demonstrated a statistically significant scale order effect among responses to a horizontal scale. My proposed experiment manipulates the alignment of the scale as a factor independent of the scale direction. Thus, I will be able to explore the interaction between scale alignment and scale

4 Page of Time-Sharing Experiments for the Social Sciences direction on survey responses. Applying the up-means-good heuristic to the use of frequency scales (Tourangeau, Couper, and Conrad, 00), I hypothesize a larger order effect for vertical presentation of the frequency scale than for horizontal presentation, i.e. a higher endorsement of the first scale points when the scale is presented vertically compared with horizontally. The final feature of this frequency scale that I will propose to vary is the vagueness of the verbal labels. Many survey researchers argue against frequency scales using vague quantifiers because the conceptual interpretation of the numerical meaning of individual scale points might vary across respondents (Schaeffer ; Tourangeau, Rips, and Rasinski, 000). By contrast, precise scale labels standardize the meaning of scale labels and are less subject to idiosyncratic interpretation. However, latest work on frequency scales using vague quantifies demonstrated that vague quantifiers perform very well in terms of accuracy and predictive validity (Al Baghal, 0; Lu et al., 00). Additionally, several large surveys (e.g., American National Election Survey, National Health Interview Survey) ask respondents to report the frequency of events, such as voting or health behaviors, on scales that use vague quantifiers. What remains unclear, however, is how the vagueness of the scale labels would moderate the scale order impact. Presumably, scales with precise labels will make the tasks of interpreting scale labels and of mapping to individual scale point more straightforward. As a result, scales with precise scale labels will be less prone to scale order effect. Accordingly, I hypothesize that frequency scales using vague quantifiers will show a bigger scale presentation order effect and that precise verbal labels will reduce the influence of scale presentation order. The third weakness of the existing research on the impact of scale presentation order is that researchers tend to only examine the resulted response distributions as a function of the scale

5 Page of presentation order. To extend the research, I propose to examine other properties of the resulting responses, such as correlation with related constructs and factor patterns. In addition, most of the existing research does not attempt to determine circumstances under which one of the three mechanisms is at work except Yan (00). She compared scale direction effect by respondents interest in the survey and concluded that the scale direction effect is caused by respondents use of anchoring and adjustment heuristics instead of satisficing. With the proposed experiment, I will focus on respondent characteristics and paradata (such as response times) that are highly indicative of respondents likelihood to satisfice and compare scale direction effect by those characteristics. These analyses will help understand which mechanism is at work for whom and under what circumstances. Study design The proposed methodological experiment comprises two sets of questions: () six items measuring mental health modeled after items from the National Health Interview Survey (NHIS) and () four health related behavioral items, both using a -point fully-labeled unipolar frequency scale. I propose a full factorial experiment crossing three factors for both sets of items (Table in Appendix A). The first factor varies the direction of response scales. For half of the sample, the scale runs from high to low. For the other half, the scale runs from low to high. The second factor varies the presentation of the response scale on a screen. The response scale is presented either horizontally or vertically. As a third factor, I am proposing to vary the vagueness of scale verbal labels, creating three versions. One version will employ only vague quantifiers. The second version shows more precise frequency labels. In the third version, a combination of both labels will be used.

6 Page of Time-Sharing Experiments for the Social Sciences In addition, I propose to add measures that can be used as validation criteria for both sets of items. A set of open-ended questions ask for a numeric report ( During an average week, how often do you? ). To avoid contamination of the answers to the two sets of questions measuring the same constructs, I propose to place the experimental items at the beginning of the questionnaire and the open-ended frequency questions at the end of the questionnaire, or vice versa. Work by Meurs and Saris () suggests that 0 minutes between the administrations of related items is sufficient to eliminate a respondent s recollection of his or her first answer when answering the second question. I will also use two GfK profile variables general health (pph00) and depression (pphdepr) to further assess criterion validity. In addition, I would be interested in analyzing respondent paradata, such as response time, click behavior, and platform used to fill out the questionnaire (web vs. mobile). Analysis plan The key independent variable for the analysis is the direction of the response scales. To analyze the main effect of scale direction, I plan to start with an examination of response distributions and variances for each individual question, and variance-covariance structure among the two sets of related survey items (mental health and health behavior) under different scale directions. To carry the research further, I propose to study the factor patterns and the predictive power of factors generated. In addition, I will use the answers to the open ended questions and the GfK profile variables to investigate validity. To explore the interaction effect of scale direction and the other scale features, I will repeat the same sets of analyses crossing scale direction and the other two features to identify scale features that induce larger scale order effect or reduce scale order effect. The findings will

7 Page of help me make recommendations on how to best design and present frequency scales for survey practitioners. To distinguish mechanisms that are at work for the scale direction effect, I will repeat the analyses by respondent characteristics to examine any subgroup differences or interactions between scale direction and respondent characteristics. I will focus on characteristics of respondents indicative of either respondent ability or respondent motivation to assess satisficing theory. I speculate that respondent ability and motivation affects the size of scale direction effect. In addition, satisficing respondents are less likely to be affected by scale alignment and vagueness of scale verbal labels whereas vagueness of scale verbal labels may affect the use of anchor and heuristics. In addition, the scale alignment may affect the use of the near-meansrelated heuristics. By combining the scale features and respondent characteristics, I hope to identify or differentiate respondents for whom different mechanisms are at work. Multi-way ANCOVA models will be fit to analyze the effect of the scale direction in combination with the two other experimental factors (scale alignment and vagueness of labels) controlling for age, gender, and education. Finally, I will examine response times to these survey items and use the results to help understand the mechanism of scale direction effect. Again, presumably satisficing respondents should take less time to answer survey questions regardless of experimental conditions whereas the use of anchor-and-adjustment heuristics takes time. Contribution to science and society The proposed experiment will allow me to examine the validity of the answers under different scale directions, effect of scale direction by scale alignment, vagueness of labels, and by respondent characteristics. This will help answering the following research questions:

8 Page of Time-Sharing Experiments for the Social Sciences Does direction of frequency response scales affect response distributions and validity of responses?. Does scale direction effect differ by scale alignment and vagueness of scale labels?. Does scale direction effect differ by respondent characteristics (e.g., education, age, gender, platform respondents used to fill out the questionnaire)?. Which scale direction produces data of better quality? The relevance of this research is underscored by a very recent discussion among survey methodologists from academia and practice on AAPORnet the official list of the American Association for Public Opinion Research (AAPOR) calling for more methodological research on the impact of scale direction. The findings from the proposed experiment will benefit the survey field by contributing new knowledge to the survey literature and providing informed practical guidelines on scale construction. Benefit for people in other scientific disciplines The findings will also benefit researchers and scientists in various other fields that rely on survey data collection, such as psychology, political science, public health, and education. They will benefit from the results about how respondents perceive response scales presented in different directions and how the direction of rating scales influence that validity of the data. Requested number of respondent-items with justification I propose an experiment with a total of 0 items. None of the proposed items includes any additional text, pictures, and audio or video clips. Therefore, the experiment comprises 0 units of length, not including the two GfK profile variables. Submitting this proposal as Special Competition for Young Investigators, 0 units of length equal a maximum of,00 respondents. The full factorial xx design leads to experimental groups, i.e., approximately

9 Page of respondents per experimental group. Power analysis shows that a total sample size of n=,0 (i.e., per cell) is needed to detect a small main effect for scale direction (f=0. based on Cohen, ) with an error probability of 0.0 and an - error probability of 0. in an ANCOVA with three covariates ignoring heterogeneous treatment effects. Holding power and error constant, a total sample size of, (i.e., per cell) is needed to detect a small effect of the two-way interaction between scale direction and vagueness of the verbal labels. Therefore, the initial total sample size for the Special Competition for Young Investigators of,00 would provide sufficient power for the proposed experiment with percent item nonresponse. See Table in Appendix E for the total sample size needed to detect small, medium, and large effects in main effects, -way interactions, and -way interactions in an ANCOVA with three covariates and an error probability of 0.0 and power of.0 and.. Power calculation done with G*Power.. (see Faul, Erdfelder, Buchner, & Lang 00).

10 Page of Time-Sharing Experiments for the Social Sciences References Al Baghal, M. T. (0). Numeric estimation and response options: An examination of the measurement properties of numeric and vague quantifier responses. Dissertation University of Nebraska-Lincoln. Belson, W.A. (). The effects of reversing the presentation order of verbal rating scales. Journal of Advertising Research,, 0-. Carp, F.M. (). Position effects on interview responses. Journal of Gerontology,, -. Cohen, J. (). Statistical power analysis for the behavioral sciences. nd ed. Hillsdale et al.: Lawrence. Christian, L.M., Parsons, N.L., & Dillman, D.A. (00). Designing scalar questions for web surveys. Sociological Methods & Research,, -. Dickson, J., & Albaum, G. (). Effects of polarity on semantic differential scales in consumer research. Advances in Consumer Research,, 0-. Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (00). Statistical power analyses using G*Power.: Tests for correlation and regression analyses. Behavior Research Methods,, -. Friedman, H.H., Herskovitz, P.J., & Pollack, S. (). The biasing effects of scale-checking styles on response to a likert scale. Proceedings of the Survey Research Methods Section, AJA, -. Keusch, F. (0). The direction of rating scales and its influence on response behavior in web surveys. Paper presented at the annual meeting of the American Association for Public Opinion Research, Orlando, FL. Krosnick, J. A. (). Survey research. Annual Review of Psychology, 0, -.

11 Page of Lu, M., Safren S. A., Skolnik, P. R., Rogers, W. H., Coady, W., Hardy, H., & Wilson, I. B. (00). Optimal recall period and response task for self-reported HIV medication adherence. AIDS and Behavior,, -. van Meurs, A. & Saris, W. E. (). Memory effects in MTMM studies. In W. E. Saris & A. van Meurs (Eds.) Evaluation of measurement instruments by meta-analysis of multitraitmultimethod studies. Amsterdam: North Holland, -. Ramstedt, B., & Krebs, D. (00). Does response scale format affect the answering of personality scales? Assessing the Big Five dimensions of personality with different response scales in a dependent sample. European Journal of Psychological Assessment,, -. Scheluga, D., Jacoby, J., & Major, B. (). Whether to agree-disagree or disagree-agree: The effect of anchor order on item response. Advances in Consumer Research,, -. Toepoel, V., Das, M., & van Soest, A. (00). Design of web questionnaires: The effect of layout in rating scales. Journal of Official Statistics,, 0-. Tourangeau, R., Couper, M. P., & Conrad, F. (00). Spacing. positioning. and order. Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly,, -. Tourangeau, R., Couper, M. P., & Conrad, F. (00). Colors. labels. and interpretative heuristics for response scales. Public Opinion Quarterly,, -. Tourangeau, R., Couper, M. P., & Conrad, F. (00). Up means good. The effect of screen position on evaluative ratings in web surveys. Public Opinion Quarterly,, -. Tourangeau, R., Rips, L. J., & Rasinski, K. (000). The psychology of survey response. Cambridge et al.: Cambridge University Press.

12 Page of Time-Sharing Experiments for the Social Sciences Tversky, A. & Kahneman, D. (). Judgment under uncertainty: Heuristics and biases. Science,, -. Weng, L, & Cheng, C. (000). Effects of response order on likert-type scales. Educational and Psychological Measurement, 0, 0-. Yan, T. (00). The Effects of the Presentation Order of Rating Scales on Survey Responses. Paper presented at the annual meeting of the Midwest Association for Public Opinion Research, Chicago, IL.

13 Page of Appendix A. Table. Experimental Factors Scale direction Scale alignment Vagueness of verbal labels high to low e.g., all of the time most of the time some of the time a little of the time never low to high e.g., never a little of the time some of the time most of the time all of the time vertical vague quantifiers horizontal B. Wording of questions for the proposed experiment all of the time most of the time some of the time a little of the time never precise frequency days or days or days or days 0 days vague quantifiers + precise frequency all of the time ( days) most of the time ( or days) some of the time ( or days) a little of the time ( or days) never (0 days Mental health items modeled after the 0 NHIS questionnaire Now we are going to ask you some questions about feelings you may have experienced during an average week. During an average week, how often do you feel a) so sad that nothing could cheer you up? b)...nervous? c)...restless or fidgety? d)...hopeless? e)...that everything was an effort? f)...worthless? Health behavior questions During an average week, how often do you a) eat foods that are high in fat and/or calories? b) eat fast food? c) eat a variety of fresh fruits and vegetables? d) do physical exercise? The original wording of the 0 NHIS question on mental health is: Now we are going to ask you some questions about feelings you may have experienced over the past 0 days. During the past 0 days, how often did you feel?

14 Page of Time-Sharing Experiments for the Social Sciences C. Wording of validation questions Open-ended mental health items During an average week, on how many days do you feel a) so sad that nothing could cheer you up? b)...nervous? c)...restless or fidgety? d)...hopeless? e)...that everything was an effort? f)...worthless? Open-ended health behavior questions During an average week, on how many days do you a) eat foods that are high in fat and/or calories? b) eat fast food? c) eat a variety of fresh fruits and vegetables? d) do physical exercise? D. GfK profile variables pph00 In general, would you say your physical health is Excellent Very good Good Fair Poor pphdepr Have you been diagnosed with any of the following medical conditions? [Depression] 0 No Yes

15 Page of E. Table. Total sample size needed to detect small (f=.0), medium (f=.), and large (f=.0) effects in main effects, -way interactions, and -way interactions in an ANCOVA with covariates and an error probability of 0.0 and power (- error probability) of.0 and.. a Power - =.0 - =. Effect size Effect size f=. f=. f=.0 f=. f=. f=.0 Main effects Scale direction (df=) 0,0 Scale alignment (df=) 0,0 Vagueness of verbal labels (df=), Two-way interactions Scale direction x Scale alignment (df=) 0,0 Scale direction x Vagueness of verbal labels (df=) 0,0 Scale alignment x Vagueness of verbal labels (df=), Three-way interaction Scale direction x Scale alignment x Vagueness of verbal labels (df=), a Sample size rounded up to allow for equal size in all groups.