Cognitive testing. Quality assurance of the survey. Ăirts Briăis Luxembourg, 13 May, 2009

Transcription

1 RIGA STRADINS UNIVERSITY Department of Public Health and Epidemiology Cognitive testing Quality assurance of the survey Ăirts Briăis Luxembourg, 13 May, 2009

2 Aim of a survey To obtain information which is Valid accurate, unbiased Reliable Sensitive Using appropriate instruments

3 Problems related to instruments In the case of questionnaires, a special problem is that their validity and reliability may be a function of the characteristics of study population (e.g., educational level) results may not be applicable to another population Validity and reliability of certain instruments may be different between populations as a function of the underlying distribution of the trait Szklo, Nieto (2007)

4 Quality assurance procedures Pretesting Pilot studies Specific task for these procedures: cognitive testing of items (questions)

5 General approach - background The focus is on survey questions Focuses on the cognitive processes Volunteer subjects are recruited, and are Interviewed in a laboratory environment The recruitment of subjects targets persons with specific characteristics of interest

6 Cognitive theory Model by Tourangeau, Rips, & Rasinski, 2000 Comprehension of the question Retrieval, recall from memory Decision, judgement Response

7 Comprehension Is the respondent following the question (interest, physical conditions)? Why the question is being asked (respondent believes)? Do and which way respondent understand terms (words, phrases, jargon)? Internal consistency of the question, matching to the close-ended ended questions. In the European context: special problem multilanguage, multicultural settings with social and economic (health care) background

8 Cognitive theory Model by Tourangeau, Rips, & Rasinski, 2000 Comprehension of the question Retrieval, recall from memory Decision, judgement Response

9 Recallability Retrieval What to remember? Is this in respondents memory at all? Recall strategy (depend on question type) Which way to remember? Judgement and believe recall Adding details

10 Cognitive theory Model by Tourangeau, Rips, & Rasinski, 2000 Comprehension of the question Retrieval, recall from memory Decision, judgement Response

11 Motivation Decision Does the respondent devote sufficient mental effort to answer the question accurately and thoughtfully? Sensitivity and social desirability Does the respondent want to tell the truth? Does he/she say something that makes him/her look "better"?

12 Cognitive theory Model by Tourangeau, Rips, & Rasinski, 2000 Comprehension of the question Retrieval, recall from memory Decision, judgement Response

13 Mapping Response Can the respondent match his answer to the response categories given by question? Can the respondent remember the given answer options? What is the interpretation of answer options for the respondent?

14 Cognitive testing methods Willis, 1999 Think-aloud interviewing Verbal probing

15 Think-aloud interviewing Respondents are asked comment on terms, unclear words and expressions terms, their understanding of the meanings Comment what they are thinking about the question, its clarity and relevance No active intervene by the interviewer Theoretically respondents should be trained Practically it is impossible (by our opinion)

16 Verbal probing Questions about the meanings, understanding, interpretation is asked by the interviewer Pre-prepared Spontaneous (more relevant by our opinion as this can be adjusted to the respondent)

17 Examples of questions By Willis, 1999 How did you arrive at this answer? Why do you believe this? [Judgment Explanation] How sure are you of your answer to this question? [Judgment Certainty] Did you compare yourself to other people or to how your health used to be in deciding how to answer this question? [Anchoring Strategy] Is this something you had already made up your mind about before we asked this question? [Attitude Crystallization]

18 Sampling Convenience sample persons Wide range of Age groups Social classes Education

19 Logical control of questions Before, during and after the cognitive testing Theoretically differs from cognitive testing In practice difficult to separate

20 To be continued... During the discussion