Questionnaire Development and Testing Barbara J. Stussman Survey Statistician NIH/NCCAM
Acknowledgments Gordon Willis, Ph.D. Applied Research Program, NCI William Riley, Ph.D. Chief, Science of Research and Technology Branch, NCI
Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire
3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods
Data Sources for Health Research Birth and death records Medical records at physician offices, hospitals, nursing homes, etc. Medical databases housed within various agencies, universities, and
insurance companies Physical exams and laboratory testing Registers of diseases Self-report measures such as interviews and questionnaires
Self-Report Measures Some information can be gathered only by asking people questions (i.e. not easily observable) Self report measures are estimates of true scores True score + Measurement error = Survey response
What are the Pitfalls of Self-Report? Susceptible to the respondent s Mood Motivation Memory Understanding
What are the Pitfalls of Self-Report? Also susceptible to: Context circumstances of interview Social desirability choosing answers that are viewed favorably Thus, importance of rigorous methods
Common Types of Questions Open-ended What health conditions do you have? Closed Which of the following conditions do you currently have? Say yes or no to each. Diabetes? Asthma? Hypertension?
Common Types of Questions Response options Nominal unordered response categories (e.g. male, female) Ordinal ordered response categories (e.g. excellent, good, fair, poor) Type of information Factual objectively verifiable facts and events Subjective knowledge, perceptions, feelings, judgment
Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire
3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Computerized Surveys
Pros Faster data availability Can handle complex skip patterns Skip Patterns Computerized Surveys
Pros Can handle complex skip patterns Can help to eliminate skip errors (but not always) Can be tailored to severity of symptoms or situation (computerized adaptive testing) Computerized Surveys
Cons Data can get lost if system crashes Requires power source In-Person Interviewer Administered Pros
Answer respondent questions, probe for adequate answers Administer to illiterate/low reading level Easier to reach poor, homeless, etc. Build rapport People feel more anonymous Can use visual aids
In-Person Interviewer Administered Cons Expensive Longer data collection period Interviewer presence/technique can bias results
Telephone Interviewer Administered Pros Lower Costs Can ensure uniform data collection Shorter data collection period Cell phones are best way to reach transient people Telephone Interviewer Administered
Cons Omit persons without phones 4 in 10 adults are cell phone only in U.S. Need complex statistical framework Cannot use visual aids Many of us do not answer our phone Paper and Pen Self- Administration
Pros Anonymity Can use longer, more complex response categories Can use visual aids Consistent across respondents Cover large geographic area Length easy to see (plus or minus?) Paper and Pen Self- Administration
Cons Good reading and writing skills required Cannot have complex skip patterns No quality control Similar cost and response rates to other methods Web/Smartphone/Tablet Self-Administration Pros Anonymity Better for sensitive items
Timely data Lower cost Can use long list of response categories Can use visual aids Any time/location Cover large geographic area Can use complex skip patterns Web/Smartphone/Tablet Self-Administration Cons
Varying degrees of computer skills, access, connection speeds, configurations Challenge to verify informed consent Concern about multiple responses from same person Difficult to track nonresponders Could be biased sample Effects of Data Collection Method
Multiple methods increase response rates but at what cost? Aural vs. Visual (Interviewer vs. selfresponse) Aural more positive How would you describe your
health, would you say excellent, good, fair or poor? Aural give more agreeable answers Effects of Data Collection Method Questions often tailored to mode
Yes/No popular with telephone; Long list of check boxes popular for web Long scales often used for selfadministered; shorter scales for telephone
Vast array of visual/graphic choices available for computerized surveys Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic
objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation
Mixed Methods Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas
2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods
Main Development Steps: 1. Determine Analytic Objectives/Topic Areas What are the general concepts to be covered/research questions? Literature review, expert panels, think tanks, patient input, focus groups, etc.
Do not forget the patient s perspective 1. Determine Analytic Objectives/Topic Areas What Type of Data Will Answer the Research Question? Outline of Lecture Self report measures Data collection methods Main questionnaire development steps
1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting
5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 2. Put Together Draft Questionnaire Use existing instruments when possible National Field Surveys (e.g. NHIS, NHANES,
National Health Service) Patient Reported Outcomes Measurement Information System (PROMIS) REDCap (Shared Library of data collection instruments) Pay attention to aesthetics
Draft new questions using known criteria How Do You Write a Question? Put together draft questionnaire Literacy < 9 th grade U.S. Specific better than broad Capturing what researcher intended Culturally sensitive Scales consistent Terms defined
How Do I Write a Question? Put together draft questionnaire Multiple concepts separated into single items Instructions clear Interpreted accurately by range of demographic groups Response options match question Reference periods clear
How Do We Write a Question? Put together draft questionnaire Avoid Social desirability effects Negative wording Double barreled Jargon Ambiguous Leading
Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire
3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Main Development Steps:
3. Cognitive Testing Cognitive Testing Technique to study the way in which respondents understand, process, and
respond to survey questions Probing techniques to determine how respondents interpret the questions Cognitive Testing
All components tested (stem, response categories, instructions, question ordering, etc.)
Qualitative analysis performed to find common themes Often performed in laboratory by trained research team with discussion of findings
Sometimes beneficial to go to the respondents (hard to reach populations) Cognitive Testing Participants based on target population
Participants start to lose focus after about 1 hour Participants usually paid Cognitive Testing Iterative Process Test questionnaire
Modify based on cognitive testing Goal Find and fix sources of Response Error Sources of Response Error
Respondent does not know the information They cannot recall it, although they do know it They do not understand the question They do not want to report the answer in the survey context
(See Fowler F. (1991). Survey Research Methods, Sage) Possible Actions Taken Based on Semi-Structured/Cognitive Interviews Accept original question
Accept original question with minor edits Accept original question with major edits Drop question/draft new question
Examples of common questionnaire problems: Too Broad Original Do you think there is scientific evidence showing that SOME alternative therapies do the following? Please say yes or no to each.
(1) Improve overall health and wellness (2) Prevent illnesses or diseases (3) Reduce pain (4) Treat or cure illnesses or diseases Examples of common questionnaire problems: Too Broad Final
Please tell me the reasons why you have never used [acupuncture, chiropractic, etc.]? [List is provided] Examples of common questionnaire problems: Too Complex Original During the past 30 days did you use any of the following
vitamins and minerals for your own health or treatment? Be sure to include ALL vitamins that you use. If you take a SINGLE vitamin or mineral supplement, such as niacin, that is not part of a combination multivitamin/mineral supplement, include it separately. Examples of common questionnaire problems: Too Complex Final
The next questions are about any vitamins and minerals you may take. Have you ever taken any vitamins or minerals listed on this card? Examples of common questionnaire problems: Double-barreled Original What was the reason you chose to use acupuncture was it to
treat a specific health problem or just to stay healthy or well? Examples of common questionnaire problems: Double-barreled Final Did you use acupuncture for any of these reasons? Please say yes or no to each. For general wellness or general disease prevention?
For one or more specific health problems, symptoms, or conditions? Examples of common questionnaire problems: Information Unknown Questions dropped Do you currently see a practitioner for homeopathy more, less, or about the same as you did one year ago?
At what age did you first start using [complementary therapy]? During the past 12 months, did your child pray for his/her own health? Examples of common questionnaire problems: Terms Undefined Original
During the past 12 months did you use movement therapies for your own health or treatment? Examples of common questionnaire problems: Terms Undefined Final Have you ever practiced any of the following movement or exercise techniques? Alexander Technique? Feldenkrais?
Pilates? Trager Psychophysical Integration? Examples of common questionnaire problems: Cultural salience Have you ever switched from a stronger to a lighter cigarette?
(Original) During the past 12 months, did you see a practitioner for/use [therapy] because it is how you were raised? (Final) During the past 12 months, did you see a practitioner for/use
[therapy] because it was part of your upbringing? Examples of common questionnaire problems: Inconsistent response options 10-fold increase in children reported to have cerebral palsy 311-353 cases in 2004-2006
34 cases in 2003 Caused by interviewer error when survey moved to new screen design Examples of common questionnaire problems: Ordering effects Original Have you ever had a sigmoidoscopy? When was your most recent?
Have you ever had a colonoscopy? When was your most recent? Examples of common questionnaire problems: Ordering effects Final Have you ever had a colonoscopy? When was your most recent?
Have you ever had a sigmoidoscopy? When was your most recent? Pay Attention to Length Ideally <= 30 minutes for face-to-face <=15 minutes for phone or web
Too long will increase costs, decrease response rates Interviewers will rush; respondents will get tired, be less careful Keystroke data have shown cheating Outline of Lecture Self report measures Data collection methods
Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common
questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 4. Field Pretesting
Realistic survey setting to evaluate process (aim for 200 cases) For Interviewer-administered surveys Experienced interviewers use near final instrument Interviewer/observer rating forms Debriefing Field Pretesting
For Self-administered surveys Interview respondents after they complete survey Observe respondents as they fill out survey Field Pretesting Tabulated data Analyze response distributions
Design closed response categories from open-ended questions Collapse/eliminate response categories Alter skip patterns Drop items Outline of Lecture Self report measures Data collection methods Main questionnaire development steps
1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting
5. Psychometrics 6. Translation Mixed Methods Main Development Steps: 5. Psychometrics Quantitative methods to statistically assess the reliability and validity of survey instruments; also a way to establish scoring mechanisms
Enables users to combine a set of items and come up with a single score (e.g. level of depression or physical functioning) Psychometrics Usually done after a large scale field test (larger instruments require larger sample)
Classical Test Theory (Old science) Requires the use of EVERY item in a set Modern Test Theory (Current standard) Focuses on contribution of each individual item in a set Psychometrics Item response theory (IRT)
Focus on contribution of SINGLE items. To what extent does each item measure the underlying construct? In the past 7 days, I felt Unhappy Never Never Never Rarely
Rarely Sometimes Sometimes Often Often Always I had no reason for living Never Never Never Never Never
Never Rarely Sometimes Often Always Psychometrics Differential item functioning (DIF) detects error related to subgroups of people
Identify items that introduce bias (Example I cry easily ) Psychometrics Karon Cook, PhD Psychometrician/Outcomes Researcher Research Professor in Medical Sciences Northwestern University Feinberg School of Medicine, Chicago, IL
Youtube video on Modern Response Theory http//www.youtube.com/watch?v=srdbllmyq8m Computerized Adaptive Testing Combines item response theory (IRT) and computer technology Question selected based on person s response to previous questions
Reduces number of questions People near top or bottom of scale receive items appropriate to them Outline of Lecture Self report measures Data collection methods Main questionnaire development steps
1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting
5. Psychometrics 6. Translation Mixed Methods 6. Translation PROMIS Approach One version for multiple countries Allows comparisons across subgroups
Translators from different regions and dialects Avoid colloquial expressions Pre-test questionnaire with participants from relevant regions Outline of Lecture Self report measures
Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of
common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods Mixed Methods Integrating or combining qualitative and quantitative
methods to draw on strengths of each Reasons for using mixed methods View problems from multiple perspectives Contextualize information Develop more complete
understanding of problem Challenges Teamwork, resources, sample size, interpretation Basic Mixed Methods Designs
Qualitative Quantitative qualitative used to develop outcome measure or intervention Quantitative Qualitative qualitative used to explain quantitative outcomes in-depth
Concurrent Qualitative used to understand participant s experiences with intervention/describ e process Mixed Methods - Examples
Nutting et al. (2002) Why are primary care physicians and nurses unable to initiate depression treatment in their practices? Interviewed 12 physicians and 6 nurse managers
Developed checklist of barriers 12 physicians completed checklist for 64 patients Cluster analysis identified groups of patients with common barrier profiles
Nutting et. al. (2002) Results: Cluster Domains Patient resistant to diagnosis or treatment Patient noncompliant with visits Physician judgment overrules guidelines Patient has psychosocial problems and limited access to care System barriers Mixed Methods - Examples
Patient Reported Outcomes Measurement Information System (PROMIS) Developed measures of patientreported health status for physical, mental, and social well-being
Expert review focus groups cognitive testing field to representative sample psychometrics Mixed Methods - Examples
Clinical practice setting Use qualitative component to understand quantitative data Study at NIH Clinical Center on persons with and without Fibromyalgia
Mixed Methods - Examples Brain imaging study where patients are given heat probes and asked to rate their level of pain Qualitative component added
to learn how patient s determine pain ratings Looking for differences between FM patients and healthy patients Suggested Resources Cognitive Interviewing (Willis, 2005/Sage)
Survey Research Methods (Fowler, 2014/Sage) Designing and Conducting Mixed Methods Research (Creswell and Clark, 2007/Wiley)
Mail and Web Surveys (Dillman, 2007/Wiley) Useful Web Resources Question Appraisal System (Willis & Lessler, 1999) (appliedresearch.ca ncer.gov/areas/
cognitive/qas99.p df) Patient Reported Outcomes Measurement Information System (PROMIS) (www.nihpromis.org )
PROMIS Assessment Center https//www.assess mentcenter.net/ Useful Web Resources Youtube video on Modern Response Theory
(www.youtube.com/watch?v= Srdbl lmyq8m) United Health Service Surveys (U.K.) (http://www.nhssurve ys.org/) Centers for Disease Control and Prevention (CDC) Surveys (www.cdc.gov/nchs/ surveys.htm)
REDCap Shared Library (www.projectredcap.org) Questions?