Questionnaire Development and Testing. Barbara J. Stussman Survey Statistician NIH/NCCAM

Questionnaire Development and Testing Barbara J. Stussman Survey Statistician NIH/NCCAM

Acknowledgments Gordon Willis, Ph.D. Applied Research Program, NCI William Riley, Ph.D. Chief, Science of Research and Technology Branch, NCI

Outline of Lecture Self report measures Data collection methods Main questionnaire development steps 1. Determine analytic objectives/topic areas 2. Put together draft questionnaire 3. Cognitive testing and examples of common questionnaire problems 4. Field pretesting 5. Psychometrics 6. Translation Mixed Methods

Data Sources for Health Research Birth and death records Medical records at physician offices, hospitals, nursing homes, etc. Medical databases housed within various agencies, universities, and insurance companies Physical exams and laboratory testing Registers of diseases Self-report measures such as interviews and questionnaires

Self-Report Measures Some information can be gathered only by asking people questions (i.e. not easily observable) Self report measures are estimates of true scores True score + Measurement error = Survey response

What are the Pitfalls of Self-Report? Susceptible to the respondent s Mood Motivation Memory Understanding

What are the Pitfalls of Self-Report? Also susceptible to: Context circumstances of interview Social desirability choosing answers that are viewed favorably Thus, importance of rigorous methods

Common Types of Questions Open-ended What health conditions do you have? Closed Which of the following conditions do you currently have? Say yes or no to each. Diabetes? Asthma? Hypertension?

Common Types of Questions Response options Nominal unordered response categories (e.g. male, female) Ordinal ordered response categories (e.g. excellent, good, fair, poor) Type of information Factual objectively verifiable facts and events Subjective knowledge, perceptions, feelings, judgment

DATA COLLECTION METHODS Interviewer Administered (Human) Self Administered Interactive Voice Response Computerized Surveys/ Electronic Data Capture In-person Telephone Web Smartphone Tablet Telephone Web Traditional Paper Surveys In-person Telephone Paper Not applicable

Computerized Surveys Pros Faster data availability Can handle complex skip patterns

Skip Patterns Have you ever had asthma? YES NO END Have you had an asthma attack in the past 12 months? YES NO END How many times have you gone to the emergency room because of asthma?

Computerized Surveys Pros Can handle complex skip patterns Can help to eliminate skip errors (but not always) Can be tailored to severity of symptoms or situation (computerized adaptive testing)

Computerized Surveys Cons Data can get lost if system crashes Requires power source

In-Person Interviewer Administered Pros Answer respondent questions, probe for adequate answers Administer to illiterate/low reading level Easier to reach poor, homeless, etc. Build rapport People feel more anonymous Can use visual aids

In-Person Interviewer Administered Cons Expensive Longer data collection period Interviewer presence/technique can bias results

Telephone Interviewer Administered Pros Lower Costs Can ensure uniform data collection Shorter data collection period Cell phones are best way to reach transient people

Telephone Interviewer Administered Cons Omit persons without phones 4 in 10 adults are cell phone only in U.S. Need complex statistical framework Cannot use visual aids Many of us do not answer our phone

Paper and Pen Self-Administration Pros Anonymity Can use longer, more complex response categories Can use visual aids Consistent across respondents Cover large geographic area Length easy to see (plus or minus?)

Paper and Pen Self-Administration Cons Good reading and writing skills required Cannot have complex skip patterns No quality control Similar cost and response rates to other methods

Web/Smartphone/Tablet Self-Administration Pros Anonymity Better for sensitive items Timely data Lower cost Can use long list of response categories Can use visual aids Any time/location Cover large geographic area Can use complex skip patterns

Web/Smartphone/Tablet Self-Administration Cons Varying degrees of computer skills, access, connection speeds, configurations Challenge to verify informed consent Concern about multiple responses from same person Difficult to track non-responders Could be biased sample

Effects of Data Collection Method Multiple methods increase response rates but at what cost? Aural vs. Visual (Interviewer vs. selfresponse) Aural more positive How would you describe your health, would you say excellent, good, fair or poor? Aural give more agreeable answers

Effects of Data Collection Method Questions often tailored to mode Yes/No popular with telephone; Long list of check boxes popular for web Long scales often used for selfadministered; shorter scales for telephone Vast array of visual/graphic choices available for computerized surveys

Main Development Steps: 1. Determine Analytic Objectives/Topic Areas What are the general concepts to be covered/research questions? Literature review, expert panels, think tanks, patient input, focus groups, etc. Do not forget the patient s perspective

1. Determine Analytic Objectives/Topic Areas What Type of Data Will Answer the Research Question? % of respondents who used yoga in past 12 months % who took a yoga class in past 12 months % who had insurance coverage for cost of yoga class Men (X%) Women (X%) (X%) (X%) (X%) (X%)

Main Development Steps: 2. Put Together Draft Questionnaire Use existing instruments when possible National Field Surveys (e.g. NHIS, NHANES, National Health Service) Patient Reported Outcomes Measurement Information System (PROMIS) REDCap (Shared Library of data collection instruments) Pay attention to aesthetics Draft new questions using known criteria

How Do You Write a Question? Put together draft questionnaire Literacy < 9 th grade U.S. Specific better than broad Capturing what researcher intended Culturally sensitive Scales consistent Terms defined

How Do I Write a Question? Put together draft questionnaire Multiple concepts separated into single items Instructions clear Interpreted accurately by range of demographic groups Response options match question Reference periods clear

How Do We Write a Question? Put together draft questionnaire Avoid Social desirability effects Negative wording Double barreled Jargon Ambiguous Leading

Main Development Steps: 3. Cognitive Testing Cognitive Testing Technique to study the way in which respondents understand, process, and respond to survey questions Probing techniques to determine how respondents interpret the questions

Cognitive Testing All components tested (stem, response categories, instructions, question ordering, etc.) Qualitative analysis performed to find common themes Often performed in laboratory by trained research team with discussion of findings Sometimes beneficial to go to the respondents (hard to reach populations)

Cognitive Testing Participants based on target population Participants start to lose focus after about 1 hour Participants usually paid

Cognitive Testing Iterative Process Test questionnaire Modify based on cognitive testing Goal Find and fix sources of Response Error

Sources of Response Error Respondent does not know the information They cannot recall it, although they do know it They do not understand the question They do not want to report the answer in the survey context (See Fowler F. (1991). Survey Research Methods, Sage)

Possible Actions Taken Based on Semi- Structured/Cognitive Interviews Accept original question Accept original question with minor edits Accept original question with major edits Drop question/draft new question

Examples of common questionnaire Original problems: Too Broad Do you think there is scientific evidence showing that SOME alternative therapies do the following? Please say yes or no to each. (1) Improve overall health and wellness (2) Prevent illnesses or diseases (3) Reduce pain (4) Treat or cure illnesses or diseases

Examples of common questionnaire problems: Too Broad Final Please tell me the reasons why you have never used [acupuncture, chiropractic, etc.]? [List is provided]

Examples of common questionnaire problems: Too Complex Original During the past 30 days did you use any of the following vitamins and minerals for your own health or treatment? Be sure to include ALL vitamins that you use. If you take a SINGLE vitamin or mineral supplement, such as niacin, that is not part of a combination multivitamin/mineral supplement, include it separately.

Examples of common questionnaire problems: Too Complex Final The next questions are about any vitamins and minerals you may take. Have you ever taken any vitamins or minerals listed on this card?

Examples of common questionnaire problems: Double-barreled Original What was the reason you chose to use acupuncture was it to treat a specific health problem or just to stay healthy or well?

Examples of common questionnaire problems: Double-barreled Final Did you use acupuncture for any of these reasons? Please say yes or no to each. For general wellness or general disease prevention? For one or more specific health problems, symptoms, or conditions?

Examples of common questionnaire problems: Information Unknown Questions dropped Do you currently see a practitioner for homeopathy more, less, or about the same as you did one year ago? At what age did you first start using [complementary therapy]? During the past 12 months, did your child pray for his/her own health?

Examples of common questionnaire problems: Terms Undefined Original During the past 12 months did you use movement therapies for your own health or treatment?

Examples of common questionnaire Final problems: Terms Undefined Have you ever practiced any of the following movement or exercise techniques? Alexander Technique? Feldenkrais? Pilates? Trager Psychophysical Integration?

Examples of common questionnaire problems: Cultural salience Have you ever switched from a stronger to a lighter cigarette? (Original) During the past 12 months, did you see a practitioner for/use [therapy] because it is how you were raised? (Final) During the past 12 months, did you see a practitioner for/use [therapy] because it was part of your upbringing?

Examples of common questionnaire problems: Inconsistent response options 10-fold increase in children reported to have cerebral palsy 311-353 cases in 2004-2006 34 cases in 2003 Caused by interviewer error when survey moved to new screen design

Looking at this list, has a doctor or other health professional ever told you that {child name} had any of these conditions? (00) None (01) Down s syndrome (02) Cerebral Palsy (03) Muscular Dystrophy (04) Cystic Fibrosis (05) Sickle cell anemia (06) Autism (07) Diabetes (08) Arthritis (09) Congenital heart disease (10) Other heart condition

Examples of common questionnaire problems: Ordering effects Original Have you ever had a sigmoidoscopy? When was your most recent? Have you ever had a colonoscopy? When was your most recent?

Examples of common questionnaire problems: Ordering effects Final Have you ever had a colonoscopy? When was your most recent? Have you ever had a sigmoidoscopy? When was your most recent?

Pay Attention to Length Ideally <= 30 minutes for face-to-face <=15 minutes for phone or web Too long will increase costs, decrease response rates Interviewers will rush; respondents will get tired, be less careful Keystroke data have shown cheating

Main Development Steps: 4. Field Pretesting Realistic survey setting to evaluate process (aim for 200 cases) For Interviewer-administered surveys Experienced interviewers use near final instrument Interviewer/observer rating forms Debriefing

Field Pretesting For Self-administered surveys Interview respondents after they complete survey Observe respondents as they fill out survey

Tabulated data Field Pretesting Analyze response distributions Design closed response categories from open-ended questions Collapse/eliminate response categories Alter skip patterns Drop items

Main Development Steps: 5. Psychometrics Quantitative methods to statistically assess the reliability and validity of survey instruments; also a way to establish scoring mechanisms Enables users to combine a set of items and come up with a single score (e.g. level of depression or physical functioning)

Psychometrics Usually done after a large scale field test (larger instruments require larger sample) Classical Test Theory (Old science) Requires the use of EVERY item in a set Modern Test Theory (Current standard) Focuses on contribution of each individual item in a set

Psychometrics Item response theory (IRT) Focus on contribution of SINGLE items. To what extent does each item measure the underlying construct?

In the past 7 days, I felt Unhappy Never Never Never Rarely Rarely Sometimes Sometimes Often Often Always I had no reason Never Never Never Never Never Never Rarely Sometimes Often Always for living

Psychometrics Differential item functioning (DIF) detects error related to subgroups of people Identify items that introduce bias (Example I cry easily )

Psychometrics Karon Cook, PhD Psychometrician/Outcomes Researcher Research Professor in Medical Sciences Northwestern University Feinberg School of Medicine, Chicago, IL Youtube video on Modern Response Theory http//www.youtube.com/watch?v=srdbllmy q8m

Computerized Adaptive Testing Combines item response theory (IRT) and computer technology Question selected based on person s response to previous questions Reduces number of questions People near top or bottom of scale receive items appropriate to them

6. Translation PROMIS Approach One version for multiple countries Allows comparisons across subgroups Translators from different regions and dialects Avoid colloquial expressions Pre-test questionnaire with participants from relevant regions

Mixed Methods Integrating or combining qualitative and quantitative methods to draw on strengths of each Reasons for using mixed methods View problems from multiple perspectives Contextualize information Develop more complete understanding of problem Challenges Teamwork, resources, sample size, interpretation

Basic Mixed Methods Designs Qualitative Quantitative qualitative used to develop outcome measure or intervention Quantitative Qualitative qualitative used to explain quantitative outcomes in-depth Concurrent Qualitative used to understand participant s experiences with intervention/describe process

Mixed Methods - Examples Nutting et al. (2002) Why are primary care physicians and nurses unable to initiate depression treatment in their practices? 1. Interviewed 12 physicians and 6 nurse managers 2. Developed checklist of barriers 3. 12 physicians completed checklist for 64 patients 4. Cluster analysis identified groups of patients with common barrier profiles

Nutting et. al. (2002) Results: Cluster Domains Patient resistant to diagnosis or treatment Patient noncompliant with visits Physician judgment overrules guidelines Patient has psychosocial problems and limited access to care System barriers

Mixed Methods - Examples Patient Reported Outcomes Measurement Information System (PROMIS) Developed measures of patient-reported health status for physical, mental, and social well-being Expert review focus groups cognitive testing field to representative sample psychometrics

Mixed Methods - Examples Clinical practice setting Use qualitative component to understand quantitative data Study at NIH Clinical Center on persons with and without Fibromyalgia

Mixed Methods - Examples Brain imaging study where patients are given heat probes and asked to rate their level of pain Qualitative component added to learn how patient s determine pain ratings Looking for differences between FM patients and healthy patients

Suggested Resources Cognitive Interviewing (Willis, 2005/Sage) Survey Research Methods (Fowler, 2014/Sage) Designing and Conducting Mixed Methods Research (Creswell and Clark, 2007/Wiley) Mail and Web Surveys (Dillman, 2007/Wiley)

Useful Web Resources Question Appraisal System (Willis & Lessler, 1999) (appliedresearch.cancer.gov/areas/ cognitive/qas99.pdf) Patient Reported Outcomes Measurement Information System (PROMIS) (www.nihpromis.org) PROMIS Assessment Center https//www.assessmentcenter.net/

Useful Web Resources Youtube video on Modern Response Theory (www.youtube.com/watch?v=srdbl lmyq8m) United Health Service Surveys (U.K.) (http://www.nhssurveys.org/) Centers for Disease Control and Prevention (CDC) Surveys (www.cdc.gov/nchs/surveys.htm) REDCap Shared Library (www.project-redcap.org)

Questions?