Questionnaire Development and Testing Barbara J. Stussman Survey Statistician NIH/NCCAM

Transcription

1 Questionnaire Development and Testing Barbara J. Stussman Survey Statistician NIH/NCCAM

2 Acknowledgments Gordon Willis, Ph.D. Applied Research Program, NCI William Riley, Ph.D. Chief, Science of Research and Technology Branch, NCI

3 Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire

4 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods

5 Data Sources for Health Research Birth and death records Medical records at physician offices, hospitals, nursing homes, etc. Medical databases housed within various agencies, universities, and

6 insurance companies Physical exams and laboratory testing Registers of diseases Self-report measures such as interviews and questionnaires

7 Self-Report Measures Some information can be gathered only by asking people questions (i.e. not easily observable) Self report measures are estimates of true scores True score + Measurement error = Survey response

8 What are the Pitfalls of Self-Report? Susceptible to the respondent s Mood Motivation Memory Understanding

9 What are the Pitfalls of Self-Report? Also susceptible to: Context circumstances of interview Social desirability choosing answers that are viewed favorably Thus, importance of rigorous methods

10 Common Types of Questions Open-ended What health conditions do you have? Closed Which of the following conditions do you currently have? Say yes or no to each. Diabetes? Asthma? Hypertension?

11 Common Types of Questions Response options Nominal unordered response categories (e.g. male, female) Ordinal ordered response categories (e.g. excellent, good, fair, poor) Type of information Factual objectively verifiable facts and events Subjective knowledge, perceptions, feelings, judgment

12 Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire

13 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Computerized Surveys

14 Pros Faster data availability Can handle complex skip patterns Skip Patterns Computerized Surveys

15 Pros Can handle complex skip patterns Can help to eliminate skip errors (but not always) Can be tailored to severity of symptoms or situation (computerized adaptive testing) Computerized Surveys

16 Cons Data can get lost if system crashes Requires power source In-Person Interviewer Administered Pros

17 Answer respondent questions, probe for adequate answers Administer to illiterate/low reading level Easier to reach poor, homeless, etc. Build rapport People feel more anonymous Can use visual aids

18 In-Person Interviewer Administered Cons Expensive Longer data collection period Interviewer presence/technique can bias results

19 Telephone Interviewer Administered Pros Lower Costs Can ensure uniform data collection Shorter data collection period Cell phones are best way to reach transient people Telephone Interviewer Administered

20 Cons Omit persons without phones 4 in 10 adults are cell phone only in U.S. Need complex statistical framework Cannot use visual aids Many of us do not answer our phone Paper and Pen Self- Administration

21 Pros Anonymity Can use longer, more complex response categories Can use visual aids Consistent across respondents Cover large geographic area Length easy to see (plus or minus?) Paper and Pen Self- Administration

22 Cons Good reading and writing skills required Cannot have complex skip patterns No quality control Similar cost and response rates to other methods Web/Smartphone/Tablet Self-Administration Pros Anonymity Better for sensitive items

23 Timely data Lower cost Can use long list of response categories Can use visual aids Any time/location Cover large geographic area Can use complex skip patterns Web/Smartphone/Tablet Self-Administration Cons

24 Varying degrees of computer skills, access, connection speeds, configurations Challenge to verify informed consent Concern about multiple responses from same person Difficult to track nonresponders Could be biased sample Effects of Data Collection Method

25 Multiple methods increase response rates but at what cost? Aural vs. Visual (Interviewer vs. selfresponse) Aural more positive How would you describe your

26 health, would you say excellent, good, fair or poor? Aural give more agreeable answers Effects of Data Collection Method Questions often tailored to mode

27 Yes/No popular with telephone; Long list of check boxes popular for web Long scales often used for selfadministered; shorter scales for telephone

28 Vast array of visual/graphic choices available for computerized surveys Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic

29 objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation

30 Mixed Methods Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas

31 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods

32 Main Development Steps: 1. Determine Analytic Objectives/Topic Areas What are the general concepts to be covered/research questions? Literature review, expert panels, think tanks, patient input, focus groups, etc.

33 Do not forget the patient s perspective 1. Determine Analytic Objectives/Topic Areas What Type of Data Will Answer the Research Question? Outline of Lecture Self report measures Data collection methods Main questionnaire development steps

34 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting

35 5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 2. Put Together Draft Questionnaire Use existing instruments when possible National Field Surveys (e.g. NHIS, NHANES,

36 National Health Service) Patient Reported Outcomes Measurement Information System (PROMIS) REDCap (Shared Library of data collection instruments) Pay attention to aesthetics

37 Draft new questions using known criteria How Do You Write a Question? Put together draft questionnaire Literacy < 9 th grade U.S. Specific better than broad Capturing what researcher intended Culturally sensitive Scales consistent Terms defined

38 How Do I Write a Question? Put together draft questionnaire Multiple concepts separated into single items Instructions clear Interpreted accurately by range of demographic groups Response options match question Reference periods clear

39 How Do We Write a Question? Put together draft questionnaire Avoid Social desirability effects Negative wording Double barreled Jargon Ambiguous Leading

40 Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire

41 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Main Development Steps:

42 3. Cognitive Testing Cognitive Testing Technique to study the way in which respondents understand, process, and

43 respond to survey questions Probing techniques to determine how respondents interpret the questions Cognitive Testing

44 All components tested (stem, response categories, instructions, question ordering, etc.)

45 Qualitative analysis performed to find common themes Often performed in laboratory by trained research team with discussion of findings

46 Sometimes beneficial to go to the respondents (hard to reach populations) Cognitive Testing Participants based on target population

47 Participants start to lose focus after about 1 hour Participants usually paid Cognitive Testing Iterative Process Test questionnaire

48 Modify based on cognitive testing Goal Find and fix sources of Response Error Sources of Response Error

49 Respondent does not know the information They cannot recall it, although they do know it They do not understand the question They do not want to report the answer in the survey context

50 (See Fowler F. (1991). Survey Research Methods, Sage) Possible Actions Taken Based on Semi-Structured/Cognitive Interviews Accept original question

51 Accept original question with minor edits Accept original question with major edits Drop question/draft new question

52 Examples of common questionnaire problems: Too Broad Original Do you think there is scientific evidence showing that SOME alternative therapies do the following? Please say yes or no to each.

53 (1) Improve overall health and wellness (2) Prevent illnesses or diseases (3) Reduce pain (4) Treat or cure illnesses or diseases Examples of common questionnaire problems: Too Broad Final

54 Please tell me the reasons why you have never used [acupuncture, chiropractic, etc.]? [List is provided] Examples of common questionnaire problems: Too Complex Original During the past 30 days did you use any of the following

55 vitamins and minerals for your own health or treatment? Be sure to include ALL vitamins that you use. If you take a SINGLE vitamin or mineral supplement, such as niacin, that is not part of a combination multivitamin/mineral supplement, include it separately. Examples of common questionnaire problems: Too Complex Final

56 The next questions are about any vitamins and minerals you may take. Have you ever taken any vitamins or minerals listed on this card? Examples of common questionnaire problems: Double-barreled Original What was the reason you chose to use acupuncture was it to

57 treat a specific health problem or just to stay healthy or well? Examples of common questionnaire problems: Double-barreled Final Did you use acupuncture for any of these reasons? Please say yes or no to each. For general wellness or general disease prevention?

58 For one or more specific health problems, symptoms, or conditions? Examples of common questionnaire problems: Information Unknown Questions dropped Do you currently see a practitioner for homeopathy more, less, or about the same as you did one year ago?

59 At what age did you first start using [complementary therapy]? During the past 12 months, did your child pray for his/her own health? Examples of common questionnaire problems: Terms Undefined Original

60 During the past 12 months did you use movement therapies for your own health or treatment? Examples of common questionnaire problems: Terms Undefined Final Have you ever practiced any of the following movement or exercise techniques? Alexander Technique? Feldenkrais?

61 Pilates? Trager Psychophysical Integration? Examples of common questionnaire problems: Cultural salience Have you ever switched from a stronger to a lighter cigarette?

62 (Original) During the past 12 months, did you see a practitioner for/use [therapy] because it is how you were raised? (Final) During the past 12 months, did you see a practitioner for/use

63 [therapy] because it was part of your upbringing? Examples of common questionnaire problems: Inconsistent response options 10-fold increase in children reported to have cerebral palsy cases in

64 34 cases in 2003 Caused by interviewer error when survey moved to new screen design Examples of common questionnaire problems: Ordering effects Original Have you ever had a sigmoidoscopy? When was your most recent?

65 Have you ever had a colonoscopy? When was your most recent? Examples of common questionnaire problems: Ordering effects Final Have you ever had a colonoscopy? When was your most recent?

66 Have you ever had a sigmoidoscopy? When was your most recent? Pay Attention to Length Ideally <= 30 minutes for face-to-face <=15 minutes for phone or web

67 Too long will increase costs, decrease response rates Interviewers will rush; respondents will get tired, be less careful Keystroke data have shown cheating Outline of Lecture Self report measures Data collection methods

68 Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common

69 questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 4. Field Pretesting

70 Realistic survey setting to evaluate process (aim for 200 cases) For Interviewer-administered surveys Experienced interviewers use near final instrument Interviewer/observer rating forms Debriefing Field Pretesting

71 For Self-administered surveys Interview respondents after they complete survey Observe respondents as they fill out survey Field Pretesting Tabulated data Analyze response distributions

72 Design closed response categories from open-ended questions Collapse/eliminate response categories Alter skip patterns Drop items Outline of Lecture Self report measures Data collection methods Main questionnaire development steps

73 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting

74 5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 5. Psychometrics Quantitative methods to statistically assess the reliability and validity of survey instruments; also a way to establish scoring mechanisms

75 Enables users to combine a set of items and come up with a single score (e.g. level of depression or physical functioning) Psychometrics Usually done after a large scale field test (larger instruments require larger sample)

76 Classical Test Theory (Old science) Requires the use of EVERY item in a set Modern Test Theory (Current standard) Focuses on contribution of each individual item in a set Psychometrics Item response theory (IRT)

77 Focus on contribution of SINGLE items. To what extent does each item measure the underlying construct? In the past 7 days, I felt Unhappy Never Never Never Rarely

78 Rarely Sometimes Sometimes Often Often Always I had no reason for living Never Never Never Never Never

79 Never Rarely Sometimes Often Always Psychometrics Differential item functioning (DIF) detects error related to subgroups of people

80 Identify items that introduce bias (Example I cry easily ) Psychometrics Karon Cook, PhD Psychometrician/Outcomes Researcher Research Professor in Medical Sciences Northwestern University Feinberg School of Medicine, Chicago, IL

81 Youtube video on Modern Response Theory http// Computerized Adaptive Testing Combines item response theory (IRT) and computer technology Question selected based on person s response to previous questions

82 Reduces number of questions People near top or bottom of scale receive items appropriate to them Outline of Lecture Self report measures Data collection methods Main questionnaire development steps

83 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting

84 5. Psychometrics 6. Translation Mixed Methods 6. Translation PROMIS Approach One version for multiple countries Allows comparisons across subgroups

85 Translators from different regions and dialects Avoid colloquial expressions Pre-test questionnaire with participants from relevant regions Outline of Lecture Self report measures

86 Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of

87 common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Mixed Methods Integrating or combining qualitative and quantitative

88 methods to draw on strengths of each Reasons for using mixed methods View problems from multiple perspectives Contextualize information Develop more complete

89 understanding of problem Challenges Teamwork, resources, sample size, interpretation Basic Mixed Methods Designs

90 Qualitative Quantitative qualitative used to develop outcome measure or intervention Quantitative Qualitative qualitative used to explain quantitative outcomes in-depth

91 Concurrent Qualitative used to understand participant s experiences with intervention/describ e process Mixed Methods - Examples

92 Nutting et al. (2002) Why are primary care physicians and nurses unable to initiate depression treatment in their practices? Interviewed 12 physicians and 6 nurse managers

93 Developed checklist of barriers 12 physicians completed checklist for 64 patients Cluster analysis identified groups of patients with common barrier profiles

94 Nutting et. al. (2002) Results: Cluster Domains Patient resistant to diagnosis or treatment Patient noncompliant with visits Physician judgment overrules guidelines Patient has psychosocial problems and limited access to care System barriers Mixed Methods - Examples

95 Patient Reported Outcomes Measurement Information System (PROMIS) Developed measures of patientreported health status for physical, mental, and social well-being

96 Expert review focus groups cognitive testing field to representative sample psychometrics Mixed Methods - Examples

97 Clinical practice setting Use qualitative component to understand quantitative data Study at NIH Clinical Center on persons with and without Fibromyalgia

98 Mixed Methods - Examples Brain imaging study where patients are given heat probes and asked to rate their level of pain Qualitative component added

99 to learn how patient s determine pain ratings Looking for differences between FM patients and healthy patients Suggested Resources Cognitive Interviewing (Willis, 2005/Sage)

100 Survey Research Methods (Fowler, 2014/Sage) Designing and Conducting Mixed Methods Research (Creswell and Clark, 2007/Wiley)

101 Mail and Web Surveys (Dillman, 2007/Wiley) Useful Web Resources Question Appraisal System (Willis & Lessler, 1999) (appliedresearch.ca ncer.gov/areas/

102 cognitive/qas99.p df) Patient Reported Outcomes Measurement Information System (PROMIS) ( )

103 PROMIS Assessment Center https// mentcenter.net/ Useful Web Resources Youtube video on Modern Response Theory

104 ( Srdbl lmyq8m) United Health Service Surveys (U.K.) ( ys.org/) Centers for Disease Control and Prevention (CDC) Surveys ( surveys.htm)

105 REDCap Shared Library ( Questions?