Questionnaire design OUTLINE Questionnaire design tests Reliability Validity POINTS TO CONSIDER Identify your research objectives. Identify your population or study sample Decide how to collect the information Personal interviews versus telephone interviews versus self administered questionnaires by mail, e-mail Design your questionnaire Questionnaire Design Questionnaire Design: Content Content Wording Layout Most questionnaires consist of five sections. 1. Identifying information 2. Demographic information 3. Outcome information 4. Exposure information 5. Information on other potential confounding variables Type of questions: Open vs. Closed questions Rule 1: Use simple non-technical words Do you think Helicobacter pylori is the causative agent for peptic ulcer in the lining of the stomach or duodenum? Rule 2: Avoid double questions Do you have vomiting and diarrhea? Do you think ulcer is caused by a bacteria? Do you have vomiting? Do you have Diarrhea?
Rule 3: Avoid questions involving negatives Are you against a ban on smoking? Do you think smoking should be banned? Rule 4: Avoid leading questions Do you agree with the majority that physical activity improves your overall health? Do you think that physical activity improves your overall health? Strongly agree Agree Neutral Disagree Strongly disagree Rule 5: Ask precise questions Did you have severe diarrhea? Have you had three or more loose bowel movements in any 24-hour period between April 25 and May 1? Rule 6: Make your response categories mutually exclusive What is your current age? 0-20 20-40 40-60 60 or more What is your current age? 0-19 20-39 40-59 60 Rule 7: Make your response categories exhaustive Car Bus Car Bus Bike Walking Rule 8: Make clear instructions (Select all that apply)
Questionnaire Design: Layout Rule 8: Make clear instructions What is the most frequent means of transportation do you use to come to school? (please select only one) Should have an introduction for the questionnaire. Should be as short as possible. The questions should flow in a logical sequence. Place the most important items in the first half of questionnaire. Lay out the questions & answer choices attractively & neatly. Be careful not to overfill the page. An ending statement is also important. Pilot test the questionnaire Test the questionnaire on a small sample of your subjects first. Can reveal unanticipated problems with question wording Helps detect any flaws in your questioning and correct these prior to the main survey. Example 1: What went wrong? What is your gross family income? Less than $20,000 $20,000 - $40,000 $40,000 - $60,000 $60,000 - $100,000 Greater than $100,000 Example 2: What went wrong? Do you or anyone in your family suffer from any chronic diseases? Other, please specify: Do not know Refuse to answer Example 3: What went wrong? How long have you lived in Canada? I was born in Canada Less than 5 years 5-10 years 11-15 years Greater than 20 years
Example 4: What went wrong? Example 5: What went wrong? What is the approximate population of the city you live in? 1,000,000 + 500,000-999,999 200,000-499,999 50,000-199,9999 Less than 50,000 Are you satisfied with the eczema treatment you are receiving from your physician? Example 6: What went wrong? Over a typical seven-day period (one week), how many times do you engage in physical activity that is sufficiently prolonged and intense to cause sweating and a rapid heart beat? At least 3 times rmally once or twice Rarely or never Example 7: Which one is better? Your current age: OR Date of birth: / / dd mm yyyy OR Your current age: 0 19 20 39 40 59 60 or more What? To early detect disease in individuals who do not show any signs of disease. Why? Reduce morbidity and mortality. How? By applying a relatively simple, inexpensive test, examination or procedure to individuals who are asymptomatic. Diagnosis: confirmation of presence or absence of a disease : is done among individuals who are not suspected of having the disease
Susceptible person Subclinical disease Clinical disease Recovery morbidity disability or death Target population Negative Negative Rescreen Diagnosis sought Positive Point of exposure SCREENING Onset of symptoms test Clinical exam Positive Intervene How do you measure screening tests? Reliability Validity Reliability refers to the repeatability or reproducibility of a test It can be assessed by repeating the test using the same or different observers Observer variation Intra observer variation Inter observer variation Estimating observer variation Percent agreement and KAPPA Reliability Example 1: 37 cases of melanoma (most serious type of skin cancer) reviewed by two pathologists Reliability Example 2: Retinopathy (eye problems that people with diabetes may have) Percent agreement= [(10+0+16)/ 37] x 100 = 70% Observed agreement= (46+32) / 100 = 78% KAPPA= 0.55 Kappa takes into consideration agreement due to chance. Kappa can be between 0 and 1 (a score above 0.4 indicates a reasonable level of agreement and above 0.6 is good).
Reliability: Abstract 1 OBJECTIVE: The authors report the first standardization of a screening instrument to identify sexual and physical abuse in a medical population. Reliability was assessed using testretest methodology. METHODS: The sample included 139 female patients in a gastroenterology clinic. RESULTS: Data indicate 81% agreement in testretest reliability of the sexual abuse instrument in comparison to the 77% agreement in test-retest reliability of the physical abuse questionnaire. Behav Med. 1995 Fall;21(3):141-50. The reliability and validity of a sexual and physical abuse history questionnaire in female patients with gastrointestinal disorders.leserman J, Drossman DA, Li Z. Reliability: Abstract 2 OBJECTIVE: The objective was to determine the reliability of the School Health Action, Planning and Evaluation System (SHAPES) physical activity questionnaire, a machine-readable questionnaire designed to collect data from all students in a school from grades 6 to 12. RESULTS: The study assessed test-retest reliability by administering the questionnaire twice, 1 wk apart to 2812 students in grades 9-12. The overall kappa coefficient for the 1-wk test-retest reliability of the questionnaire items indicated moderate agreement (mean 0.57 +/- 0.24). Med Sci Sports Exerc. 2006 Sep;38(9):1593-600. Reliability and validity of a school-based physical activity questionnaire. Wong SL, Leatherdale ST, Manske SR. Validity Is the test measuring what it is supposed to measure? Sensitivity Specificity Predictive values: Positive Predictive Value (PPV) Negative Predictive Value (NPV) Assumes a Gold Standard is present so as to compare it to the test Validity Status Gold Standard Test Results Yes No Total +ve a (TP) b (FP) a+b - ve c (FN) d (TN) c+d TP= True positive; FP= False positive TN= True negative; FN= False negative Sensitivity Sensitivity: The probability that the test will be positive if the disease is present. The ability of a test to correctly identify those who have a disease Sensitivity = a/a+c A highly sensitive test will have few false negatives Specificity Specificity: The probability that the test will be negative if the disease is truly absent. The ability of a test to correctly identify those who do not have the disease. Specificity = d/b+d A highly specific test will have few false positive
Predictive values Measure the probability of having the disease given the results of the test. Positive predictive values Positive predictive value: The probability of truly having the disease when a screening test is positive. Status Gold Standard Test Results Yes No Total +ve a (TP) b (FP) a+b - ve c (FN) d (TN) c+d PPV / NPV PPV = a/a+b Sensitivity / Specificity Negative predictive values Validity: Example 1 Negative predictive value: The probability of truly not having the disease when the screening test is negative Mammography Breast cancer Confirmed Not confirmed Total + 132 983 1115 _ 45 63650 63695 Total 177 64633 64810 TP= 132 Sensitivity = 132/177= 75% TN= 63650 Specificity= 63650/64633= 98% NPV = d/c+d FP= 983 PPV= 132/1115= 11.8% FN= 45 NPV= 63650/63695= 99.9% Validity: Example 2 5000 women underwent a test for blood glucose at 24 weeks of gestation. The following results were noted: Gestational diabetes Blood glucose Yes No Total 6.8 mmol/l 186 57 243 6.8 mmol/l 4 4753 4757 Total 190 4810 5000 TP= 186 Sensitivity = 186/190= 97.9% TN= 4753 Specificity= 4753/4810= 98.8% FP= 57 PPV= 186/243= 76.5% FN= 4 NPV= 4753/4757= 99.9%