BEST PRACTICES FOR RESEARCH SURVEY SELECTION, DEVELOPMENT, AND EVALUATION. Michelle A. Mengeling, PhD

Transcription

1 1 BEST PRACTICES FOR RESEARCH SURVEY SELECTION, DEVELOPMENT, AND EVALUATION Michelle A. Mengeling, PhD

2 2 There is an increased awareness of the impact of health and health care on the quality of human life. Streiner & Norman Health Measurement Scales: A practical guide to their development and use

3 3 Streiner & Norman Health Measurement Scales: A practical guide to their development and use Methods must be devised to assess in a reproducible and valid fashion those subjective states, which cannot be converted into the position of a needle on a dial.

4 4 Resilience, Quality of Life Anxiety, Depression Teamwork, Trust Preferences Intentions Evaluations

5 5 Purpose: What do you want to measure? (and who?)

6 6 Example Benefits of yoga: treatment of chronic low back pain boost immunity stress management reduce risk for disease (e.g., cardiovascular disease, diabetes) Do physical therapists use yoga in clinical practice and how?

7 7 Specifying the Research Objective What: What is yoga? What constitutes a yoga practice? Does type of yoga matter? (e.g., Hatha vs. Tantra vs. Bikram) How: Minimum number of sessions or length of sessions? Patient population? When introduced in the therapy protocol? Who: Physical therapists

8 1 Select or Create? Use an existing instrument? Develop your own?

9 9 Benefits of using an Existing Survey Comparisons across studies Documented psychometric properties (e.g., reliability & validity) Time and cost considerations

10 10 Choosing an Existing Measure Considerations: 1. Purpose 2. Number of items 3. Reliability 4. Validity 5. Cost 6. Appropriate populations 7. Mode of administration

11 Comparing Two Self-Administered Scales Purpose Beck Depression Inventory-II (BDI-II) Depression symptoms Corresponds to DMS-IV criteria Minimal, mild, moderate, severe # of items 21 multiple choice 9 Likert items Time 5-10 minutes 2 minutes Reliability Validity Internal consistency, item-scale correlations, sensitive to change Correlates with clinician assessments 11 Patient Health Questionnaire (PHQ-9) Corresponds to DMS-IV criteria + psychosocial impairment No depression, mild, moderate, moderately severe, severe Test-retest, internal consistency, sensitive to change Correlates with clinician assessments, SF-20, disability days Populations Ages 13-80; Variety Ages 19 and older; Variety Cost Yes No Use agreement

12 12 Developing a Survey Considerations Can t find a scale to measure what you d like to measure Expertise content, measurement Time and cost considerations

13 13 Survey Process Biemer PP & Lyberg LE. Introduction to Survey Quality

14 14 Research Objective(s) Specify the key research questions that the survey will be designed to answer.

15 15 Research Question to Survey Development Research Question Are women veterans health care preferences and perceptions associated with VA health care use? Survey Sociodemographics Military Characteristics Health status Trauma history Preferences: choice of provider Perceptions: VA serves needs of male and female Veterans equally well

16 Item content coverage Identify existing questions/surveys Conduct focus groups with target population Conceptual models Example: Theory of Planned Behavior Literature review Content experts

17 17 Research Objective(s) Good practice: Link survey questions back to research questions. Research Question Item 1 Item 2 Item 3 Item 4 Item

18 18 Target Population Inclusion criteria Exclusion criteria The target population represents the group who you will be able to generalize the survey results to.

19 19 Population Example Sampled from Population: Servicewomen who had served after Oct. 1, 2001 Active duty (still serving) and veterans Army or Air Force Midwestern states (IA, IL, MO, NE, KS) Exclusion Criteria: hearing or comprehension difficulties that interfered with independent telephone interviews dishonorable discharge Limitations: Regional versus national sample of servicewomen Possible limited generalizability to other racial or ethnic minorities

20 20 Sampling Types Benefits of Simple Random Sampling Sample is representative of the population Can use the statistical methods taught in basic statistics classes Can estimate precision of the estimates Easy to apply Reproducible method

21 21 Sampling Types Other Options Nonrandom sampling (e.g., snowball, purposive, convenience) Systematic sampling Stratified random sampling (weights) Multistage and cluster sampling (e.g., classrooms)

22 22 Mode of Administration Examples: Self-administered (e.g., paper and pencil, online) Telephone surveys (Computer-assisted telephone interviews) Face-to-face interviews (transcription costs) Considerations: Participant recruitment/ Response rates Missing data Skip patterns Costs

23 23 Survey Questions Goals: 1. Question is interpreted by the respondent exactly as the researcher intended 2. Questions should be efficient (i.e., minimize error & reduce participant burden) 3. Data collection costs match budget

24 24 Planning for Data Collection and Processing The goal is to minimize all sources of measurement error.

25 25 Potential Sources of Measurement Error Interviewers Respondents Mode of data collection Questionnaire

26 26 Interviewers Benefits Potentially increase participation Interpret or clarify questions Ask follow-up probes to clarify responses Provide additional data through their observations

27 27 Interviewers Interviewer variability (good vs. bad) Minimizing error: Training Standardized interviewing ( scripts ) Assessing variability/interviewer quality

28 28 Respondents Potential sources of error introduced by the Respondent Understanding the question Remembering Recall errors Telescoping error (date of event is inaccurate) Response options Informed by the multiple choice options Left open-ended (recommended for behavioral frequencies) Social desirability Fear of disclosure Respondent tells you what you want to hear

29 29 Mode of Administration Potential Error Sources: Telephone interviewing Have to have a telephone to be included Face-to-face interviewing Potentially more interviewer variability Social desirability bias Mail surveys Low response rates Nonresponse person not interested? Lost in the mail? Who completed the survey? May take longer to carry out

30 30 Questionnaire Minimizing Survey Error Pretesting Expert review of survey items Have representatives from the target population review items Behavior coding (e.g., how often does the interviewer change the wording of the question or how often does the respondent request clarification) Cognitive interviewing (e.g., comprehension, remembering)

31 31 Survey Evaluation Nonresponse Bias Initial DMDC Sample 3,245 Active Component (AC) 2,894 Reserve and National Guard (RNG) Deployed (unable to interview) 125 AC 56 RNG Unable to contact 759 AC 1,200 RNG Asked to participate 2,355 AC 1,618 RNG Ineligible 5 AC 17 RNG Deceased 1 AC 3 RNG Recruitment suspended 724 AC 462 RNG Agreed to participate 765 AC 832 RNG RESPONSE RATE 57% Refusals 866 AC 350 RNG AC (765/ )) RNG (806 /( )) Maximum interviews reached 57 AC 120 RNG Completed interviews 674 AC 665 RNG Completed partial interview 34 AC 21 RNG

32 32 Survey Evaluation: Bias Response rate of 63% Most common reason for refusing participation: too busy 615 declined participation, 64% agreed to answer standard questions by telephone No significant differences between participants and refusers: average age (38 vs. 38 years old) self-report of very good or excellent health (44% vs. 45%) number of gynecologic visits in last year (2.1 vs. 1.7) or ever being told by a provider they had an abnormal Pap test (57% vs 51%)

33 33 Survey Evaluation: Embedded Experiments Reimbursement Effect on Participation Rates Phase 1: Highly rural Veterans were more likely to participate when offered compensation (9% vs. 4%, p = 0.03); no difference for rural Veterans (7% vs. 5%, p = 0.23). In Phase 2: participation rates were similar for rural (6%), highly rural (4%), and urban (6%) (p = 0.31). Lots of literature available on participant reimbursement/compensation. Not yet published.

34 34 Survey Evaluation: Embedded Experiments vs. Snail+ VS. 15% 24% Not yet published.

35 35 Survey Evaluation: Embedded Experiments Item A Which of the following best describes your current sexual preference? Item B Which of the following best describes your sexual partners? Heterosexual Lesbian Bisexual Don t Know (n=0) Refuse (n=8) Men only Women only Both men and women No partners Don t know (n=0) Refuse (n=0)

36 36 Survey Evaluation Trust but Verify Example: Self-report (survey) vs. Chart review data 91% agreement between participants self-report of having a Pap test at a VA within the past 5 yrs and chart findings 82% agreement between both (1) having a Pap test at the VA within the last 5 years and (2) reporting results correctly as abnormal vs. normal

37 Reliability and Validity Reliability consistency of the measurements Validity how well an instrument measures what it claims to measure

38 Reliability There is no such thing as the reliability of a test, unqualified. Reliability is relative. VS.

39 Reliability: Coefficient Alpha Cronbach's alpha determines the internal consistency or average correlation of items in a survey instrument to gauge its reliability. Easy to compute in common statistical packages and spreadsheet software.

40 Reliability This test is very reliable. Reliability refers to the results obtained with an evaluation instrument and not the instrument itself. Gronlund and Linn (1990)

41 Tangent: Power & Sample Size Knowing the reliability of a scale can help a researcher estimate the sample size needed to show a statistically significant effect (power). Reliability Initial Sample size Final Sample Size *(1/0.80) = *(1/0.70) = 143

42 Improving Reliability 1. Increase the magnitude of the variance between subjects 2. Reduce the error variance

43 Validity Demonstrating an instrument is valid is really a process of gathering evidence ( validity evidence) to support the inferences we make from the resulting data. You can never prove that a scale is valid, you can only provide different and multiple types of evidence to support a scale s validity.

44 ITEM WRITING 44

45 45 Demographics Recommend not recreating the wheel.

46 46 Sources: Government Publications National Survey of Family Growth NSFG FEMALE CRQ FOR OMB Number (exp 4/30/15) Q: Who, if anyone, do you think of as the woman who mostly raised you when you were growing up? [If there is more than one woman R considers raised her, and they are equally important, probe for parent-figure during the teen years.] Biological mother...1 Adoptive mother...2 Step-mother...3 Father's girlfriend...4 Foster mother...5 Grandmother...6 Other female relative...7 Female nonrelative...8 No such person...9 Other...10

47 47 Current Marital Status Q: Now I'd like to ask about marital status and living together. Please look at Card 1. What is your current marital or cohabiting status? [Enter [2] if R is living together with a partner of the opposite sex to whom she is not married, even if she is also widowed, divorced, separated, or never-married ] [IF R volunteers living in a same-sex marriage or with a same-sex partner, then enter this information in an F2 comment.] 1. Married to a person of the opposite sex 2. Not married but living together with a partner of the opposite sex 3. Widowed 4. Divorced or annulled 5. Separated, because you and your spouse are not getting along 6. Never been married

48 48 Current Marital Status What is your current marital status? [READ RESPONSES] 1. Single, never married 2. Married, living with spouse 3. Married but separated 4. Married but geographically separated 5. Living with a partner 6. Divorced 7. Widowed 77. Don t know 99. Refuse

49 49 Race What is Person 1 s race? Mark (X) one or more boxes. White Black or African Am. American Indian or Alaska Native Print name of enrolled or principal tribe. Asian Indian Japanese Native Hawaiian Chinese Korean Guamanian or Chamorro Filipino Vietnamese Samoan Other Asian Print race, for example, Hmong, Laotian, Thai, Pakistani, Cambodian, and so on. Other Pacific Islander Print race, for example, Fijian, Tongan, and so on. Some other race Print race.

50 50 No Right or Wrong Answer Appraisal of current health Satisfaction with health care provider Health care preferences Risk assessments Opinions

51 Developing Items General Guidelines Reading load/vocabulary (e.g., most newspapers are written at a 6 th -grade reading level) Should be unambiguous Avoid double barreled items Avoid ambiguous pronoun references Avoid unnecessary wordiness Inclusion of negatively and positively worded items

52 52 Survey Questions: Example 1 How would you rate your health?

53 53 Survey Questions How would you rate your health? Excellent Very good Good Fair Poor

54 54 Survey Questions How would you rate your health? 10 Best 0 Worst

55 55 Survey Questions How would you rate your health? Good Not Good

56 56 Benefits and Drawbacks More response options provide richer data (more information) 60% 80% VS. 20% 5% 15% 20% Never Ever

57 57 Benefits and Drawbacks More response options provide richer data (more information) Little additional information can be provided when more than 10 categories are used 5-7 categories are commonly used

58 58 Considerations Administration Type Self-administered In-person By telephone Response options Numbers Labels

59 59 Example 2: I. Will you buy a new car within the next 12 month? Yes No II. How likely are you to buy a new car within the next 12 months? Not Very Likely Very Likely

60 60 Example 3: Assess hotel guest satisfaction with a 3-item survey at the time of check out. Survey 1 Survey 2 1. Did you enjoy your stay? Yes.1 No How good was the service at the hotel? Good.1 Poor Would you stay here again? Yes.1 No Overall, considering the service, food, and all other aspects of our hotel, how would you describe your stay here? Very enjoyable 1 Somewhat enjoyable 2 Neither enjoyable or unenjoyable 3 Somewhat unenjoyable 4 Very unenjoyable 5 2. How would you describe our service? More efficient than other hotels I have stayed in 1 Equally efficient to other hotels I have stayed in 2 Less efficient than other hotels I have stayed in 3 3. How likely are you to stay here again? Highly likely 1 Likely 2 Not sure 3 Unlikely 4 Highly unlikely 5

61 61 Example 4: 14. What is your religious preference? SELECT ONE 1. Protestant 2. Catholic 3. Jewish 4. Islamic 5. Something else 6. None 77. Don t know 99. Refuse [If <5>, go to 15] 14a. What is that? (fill in)

62 62 Example 5: Survey 1 Survey 2 1. Did you enjoy your stay? Yes.1 No How good was the service at the hotel? Good.1 Poor Would you stay here again? Yes.1 No Overall, considering the service, food, and all other aspects of our hotel, how would you describe your stay here? Very enjoyable 1 Somewhat enjoyable 2 Neither enjoyable or unenjoyable 3 Somewhat unenjoyable 4 Very unenjoyable 5 2. How would you describe our service? More efficient than other hotels I have stayed in 1 Equally efficient to other hotels I have stayed in 2 Less efficient than other hotels I have stayed in 3 3. How likely are you to stay here again? Highly likely 1 Likely 2 Not sure 3 Unlikely 4 Highly unlikely 5

63 63 Structure of Closed Survey Items Introductions Motivation Information regarding the content Information regarding a definition Instruction of the respondent Instructions for interviewer Request for answers (i.e., the question) Response options Saris WE & Gallhofer IN. Design, Evaluation, and Analysis of Questionnaires for Survey Research

64 64 Survey Item Introduction Examples: I am going to start with some questions about your military service. Now I would like to ask you a few questions about your current health and healthcare. These next questions ask about your use of caffeine.

65 65 Survey Item Motivation Example: It is important that the research be representative for the entire population. In order to obtain this, we need to know the general range of incomes of all people whom we interview. Saris WE & Gallhofer IN. Design, Evaluation, and Analysis of Questionnaires for Survey Research

66 66 Survey Item Content Information Explains something about the content of the item. Example: As you probably know some politicians fear that within a few years they will have to deal with an energy shortage. They therefore propose building more nuclear reactors. Other politicians warn about the dangers of nuclear energy and therefore suggest that no new reactors should be built. Saris WE & Gallhofer IN. Design, Evaluation, and Analysis of Questionnaires for Survey Research

67 67 Survey Item Definition Information Examples: [These next questions ask about your use of caffeine.] Keep in mind that a can of soda pop and a medium cup of coffee or tea is approximately 12 oz. [Has a health care clinician ever diagnosed you with Agoraphobia,] that is, anxiety about being in places or situations from which escape might be difficult or help might not be available, such as being outside the home alone, in a crowd or line, on a bridge, or traveling?

68 68 Survey Item Respondent Instructions Examples: [Now I am going to read a list of activities that you might do during a typical day.] As I read each item, please tell me if your health now limits you a lot, limits you a little, or does not limit you at all in these activities. In the past 5 years, did you receive all, some, or none of your medical care at Veteran Affairs Medical Centers?

69 69 Survey Item Interviewer Instructions Examples: INTERVIEWER: IF IN ANOTHER BRANCH NOT LISTED - SUPRESS INTERVIEWER: ASK FOR ESTIMATE IF SUBJECT IS UNCERTAIN OF EXACT DAY OR MONTH

70 70 Survey Item Request for Answers Example: Q: What is the highest level of educational training that you finished with a degree? 1. High School/GED completion 2. Some college or technical training (including junior college, technical degree, or 3-year R.N. degree) 3. College completion (4-year degree) 4. Some graduate education (at least 1 year) 5. Graduate or professional degree completed (Answer categories)

71 71 Basics Survey Item = Question + Response Options All the rest is to support the inferences from the data collected with survey methods.

72 72 Recommended Reading Internet, Mail and Mixed-mode Surveys: The Tailored Design Method by Dillman, Smyth, Christian Design, Evaluation, and Analysis of Questionnaires for Survey Research by Saris & Gallhofer Asking Questions: The Definitive Guide to Questionnaire Design For Market Research, Political Polls, and Social and Health Questionnaries by Bradburn, Sudman & Wansink Introduction to Survey Quality by Biemer & Lyberg Thinking About Answers: The Application of Cognitive Processes to Survey Methodology by Sudman, Bradburn, & Schwarz

73 73 Recommended Reading Handbook of Health Survey Methods Edited by Johnson Survey Research Methods by Fowler, Jr. Reliability for the Social Sciences: Theory and Application by Traub International Handbook of Survey Methodology by de Leeuw, Hox, & Dillman Health Measurement Scales: A Practical Guide to their Development and Use by Streiner & Norman The Psychology of Survey Response by Tourangeau, Rips, & Rasinki

74 Thank you. 74