Model-based Diagnostic Assessment. University of Kansas Item Response Theory Stats Camp 07

Transcription

1 Model-based Diagnostic Assessment University of Kansas Item Response Theory Stats Camp 07

2 Overview Diagnostic Assessment Methods (commonly called Cognitive Diagnosis). Why Cognitive Diagnosis? Cognitive Diagnosis Models. Uses of Cognitive Diagnosis.

3 Why Cognitive Diagnosis? Imagine that an elementary teacher wants to test basic math ability. Using traditional test theories the teacher could estimate an ability or test score. By knowing each examinee s score, the students can be ordered along a continuous scale. We know who is doing well and who is not.

4 Why Cognitive Diagnosis? As an alternative, we can express math ability as a set of basic skills. Addition Subtraction Multiplication Division Cognitive Diagnosis Models do not assign a score. Instead, a profile of mastered skills is estimated.

5 Why Cognitive Diagnosis? Cognitive diagnosis models are a special case of latent class models. Classes are defined by mastery of a set of dichotomous skills. Cognitive diagnosis models provide WHY students are not performing well instead of only specifying WHICH students are not doing well.

6 Why Cognitive Diagnosis? Using Traditional Scoring Using Cognitive Diagnosis Has a score of 20. Has a 75% C. Is in the 60 th percentile of math. Has scored above the cut off, passes math. Is proficient using addition. Is proficient using subtraction. Should work on Multiplication. Should work on Division.

7 Cognitive Diagnosis Models Because of the fine grained information Cognitive Diagnosis can be useful in other situations. General psychological assessment. Psychiatric diagnosis. Proficiency assessment. Before showing an example, I will further define these models.

8 Cognitive Diagnosis Models Let s revisit the idea of a basic math test. Possible items may be: /2 (4 x 2) + 3 Not all items measure all skills. A Q-matrix is used to indicate which skills are required for each item.

9 The Q-Matrix An example Q-matrix using our math test. Add Sub Mult Div / (4 x 2)

10 Examinees Examinees are characterized by profiles specifying which skills have been mastered. This is done in the same way as the Q-matrix. For example: Add Sub Mult Div Examinee A

11 Examinees Q-matrix Add Sub Mult Div / (4 x 2) Examinee Mastery Add Sub Mult Div Examinee Examinee Examinee Examinee By knowing what skills are required by each item and what skills have been mastered by an individual, we can determine which items will likely be answered correctly by each examinee. Prob Ans #1 Prob Ans #2 Prob Ans #3 Prob Ans #1 & #3

12 Cognitive Diagnosis Models The models define the chances of a correct response based on the examinee s profile. The models range in their complexity. Each model determines the basic set of assumptions that relate a person s profile to how they are expected to respond.

13 Cognitive Diagnosis Models First a little more notation I is the total number of examines J is the total number of items Q-matrix is an indicator matrix of which attributes are required for each item α i is the mastery profile for the i th examinee In describing the models we will focus on the item: 2+3-1=?

14 The DINA Model One of the simpler models Divides people into two groups (ξ ij ) Have mastered all required attributes Have not mastered all required attributes Defines probability for those who have mastered all required attributes (1-s j ). Defines probability for those who have not mastered all required attributes (g j ).

15 The DINA Model The DINA Model Assumes that you must have all skills to have a good chance at a correct response. Lacking 1 skill is the same as lacking all required skills. P( X ij = 1 ξ ij ) = (1 s j ) ξ ij g (1 ξ j ij )

16 The DINA Model Example DINA Probabilities P(Correct Response) s j =0.1 g j = (1,1,, ) (1,0,, ) (0,1,, ) (0,0,, ) Attribute Pattern

17 Reduced Reparameterized Unified Model As an alternative, the Reduced Reparameterized Unified Model (RRUM) allows each attribute to contribute differently to the probability of a correct response Uses π * to indicate the probability of a correct response if all attributes are mastered. The value r jk* is used to define the penalty for not mastering the k th attribute

18 Reduced Reparameterized Unified Model RRUM Drops the chances of a correct response based on the skill that is lacked. P( X = 1 α ) = π r ij i * j * jk q jk (1 α ik )

19 Reduced Reparameterized Unified Model Example RRUM Probabilities π * =.9 r j1 *=.444 r j2* = (1,1,, ) (1,0,, ) (0,1,, ) (0,0,, ) This implies that knowing addition is more important than knowing subtraction for the problem 2+3-1=?

20 Applications in Education Their use in education applies mostly to lowstakes tests. For example, these models are being used by the North Carolina Partnership in Improving Math and Science (NC-PIMS), a grant founded by NSF and DOE. Student specific instruction can improve students performances in preparation for the end-of grade test.

21 Applications in Psychology However, their value reaches beyond only educational assessment. For example, in psychology, many psychological disorders are defined based on satisfying a set of criteria. I will use Pathological Gambling as an example.

22 Applications in Psychology Accessibility of Gambling has increased exponentially. With this increase, the incidence of pathological gambling has also increased. Pathological Gambling is defined by the DSM-VI-TR as an impulse control disorder not otherwise classified

23 Pathological Gambling To be classified as a pathological gambler an individual must meet 5 of 10 criteria: Criterion 1 (C1): Is preoccupied with gambling. Criterion 3 (C3): Has repeated unsuccessful efforts to control, cut back, or stop gambling. Criterion 6 (C6): After losing money gambling, often returns another day to get even.

24 The Gambling Research Instrument The Gambling Research Instrument (GRI) was developed to study Pathological Gambling. It was not originally developed for diagnosis. However, we will be able to use a cognitive diagnosis model. Contained 41 Likert scale items (strongly agree-strongly disagree). Examples include: I would like to cut back on my gambling. I find it difficult to stop gambling.

25 The Gambling Research Instrument I would like to cut back on my gambling. (C1) Preoccupied with gambling (C2) Needs to gamble with increasing amounts (C3) Unsuccessful at stopping or cutting back (C6) Returns to win back money I find it difficult to stop gambling. (C3) Unsuccessful at stopping or cutting back (C4) Restless when trying to cutback or stop

26 A New Model One problem is that for any item, it seems like if you meet any one of the criteria measured by that item you would respond in a positive way I find it difficult to stop gambling. (C3) Unsuccessful at stopping or cutting back (C4) Restless when trying to cutback or stop So we developed a model based on the DINA called the DINO. If an individual meets as least required attribute they are likely to respond positively.

27 Sample Characteristics Included 128 experienced gamblers recruited from one of three methods. A mid-western casino. A letter to VIP members. An ad in select local papers. Each person was asked to complete the GRI and the South Oaks Gambling Screen (Leisure & Blume, 1987) In developing the SOGS, great effort was placed on validation A score of 5 or higher is a probable pathological gambler

28 Analysis The GRI was analyzed with new model discussed using Markov Chain Monte Carlo estimation. For each examinee the results include: The chances each criterion was met. The chances that at least 5 were met (the definition of pathological gambling). Estimates of item parameters. SOGS Score and SOGS Classification.

29 Results Discuss individuals estimates. Validate method by comparing the probability of a pathological gambler with the SOGS classification. Discuss conclusions that can be made from the estimated item parameters.

30 Individual A Probability of Satisfying Criteria C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Individual A has a 0% chance of meeting 5 or more criteria.

31 Individual B Probability of Satisfying Criteria C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Individual A has a 41% chance of meeting 5 or more criteria.

32 GRI versus SOGS SOGS Classification Classification Non-PG PG Total CDM Classification Non-PG PG Total % matching classification. Cohen s Kappa 0.69 (p-value<.001).

33 Item Parameters The item parameters can provide useful information to which items are good and which items are bad. Defined as the mean difference between those who should positively address the item and those who should not. For example, the best item had a difference of 2.84 and the worst item had a difference of 0.27.

34 Application in Psychology Using pathological gambling as an example, I have shown the advantages of cognitive diagnosis models in that: A diagnosis of pathological gambling could be obtained. Criteria level information for an individual can be gained, which can be used to focus on treatments.

35 Application in Psychology Other work has explored the reasonableness of the 5 or more criteria defined by the DSM. Future studies should validate examinee profiles. Use other methods of classification. Use a larger sample.

36 Conclusion There are many different situations where cognitive diagnosis models can be useful. Specifically, in those cases where: We have dichotomous skills. When classification is the goal. In those cases, cognitive diagnosis models can provide detailed information about each person. The information can be used to improve our treatment for each examinee.