The Psychometric Development Process of Recovery Measures and Markers: Classical Test Theory and Item Response Theory

Transcription

1 The Psychometric Development Process of Recovery Measures and Markers: Classical Test Theory and Item Response Theory Kate DeRoche, M.A. Mental Health Center of Denver Antonio Olmos, Ph.D. Mental Health Center of Denver Susan Hutchinson, Ph.D. University of Northern Colorado CJ McKinney, M.A. Mental Health Center of Denver

2 What presentation is this? We plan to combine 2 presentations: The Psychometrics of a Self-Report Recovery Measure: A Comparison of Classical Test Theory and Rasch Modeling Mental Health Center of Denver s (MHCD) Recovery Markers: Development of a Clinician Based Measure of Recovery From Mental Illness into one presentation with 2 examples (scales under development) We will focus on the process for the development of two measurement tools, including advantages and limitations of psychometric methodologies: 1 st Classical Test Theory 2 nd Rasch Models in IRT 3 rd Additional Models in IRT

3 Presentation Overview Warning: We have a wealth of data to discuss Overview: Briefly describe CTT and IRT Recovery initiative at MHCD and its implications for outcomes measures Example 1: Consumer Recovery Measure CTT Analysis Rasch Model Example 2: Recovery Markers CTT Analysis Partial Credit Model Nominal IRT Model Lessons Learned How many of you have used Item Response Theory, or have a background knowledge of how it works?

4 Comparison of CTT and IRT (Hambleton, Swaminathan, Roger, 1991) Classical Test Theory (CTT) Item characteristics are sample and test dependent Items are commonly at an equal level of the trait Reliability and assumption of paralleltest, difficult to obtain Equal standard error for all participants Item Response Theory (IRT) Separation of item parameters and participants ability Items are monotonically increasing in the latent trait Assumptions of unidimensionality & local independence Multiple models (1PL, 2PL & 3PL)

5 Models of CTT and IRT CTT has a single model IRT includes a collection of models (validity issues in model selection) 1PL (Rasch Models) N=100 s Difficulty parameters 2PL N= 1,000 s Difficulty and item discrimination parameters 3PL N= 10,000 s Difficulty, item discrimination, & pseudo guessing parameters

6 The Recovery Initiative at MHCD and its implications for outcomes Measurement The Mental Health Center of Denver is large, nonprofit community-based mental health center providing services for adults, children and families In the Evaluation and Research Department, we hold the belief that models should be data driven Evaluation of Recovery through 3 measures: Consumer Recovery Measure Recovery Markers Recovery Enhancing Environment Recovery Marker We have created 2 instruments to evaluate mental health recovery (latent trait) in adults consumers from two different perspectives Recovery Enhancing Environment Mental Health Recovery as defined at MHCD Consumer Recovery Measure

7 Theory and Measurement The relationship between theory and measurement is critical for latent constructs to be able to provide a feedback loop for quality improvement Measurement revised the underlying constructs of a theory which, in turn, revises the measurement tool Continuous Process Improve Theory Improve Theory Initially Define Theory Improve Measures Improve Measures Develop Measures

8 Example 1: Consumer Recovery Measures Measures recovery from the consumer s point of view in 5 domains: Active/growth orientation Hope Symptom s interference Safety Social network

9 Step 1: Classical Test Theory Conducted an Exploratory Factor analysis which revealed 5 factors and explained approximately 57% of the variance Conducted Cronbach s alpha reliability analysis Active/growth orientation (α = 0.67) Hope (α = 0.77) Symptom Interference (α = 0.88) Safety (α =.72) Social Network (α = 0.63) Total Scale = 0.88 According to this statistics we should conclude that the scale is not bad.

10 Step 2: Rasch Modeling Rating Scale Model (1PL) In IRT, a validity issue is selecting a model that is appropriate for your data. Most commonly, you begin with the simplest model (Rasch Model) and if it fits, you can stop, if not you can try a more complicated model. (some people do not agree with this concept) For example, our data is a Likert-type scale so we used a Rasch Rating Scale Model, which produced the following reliabilities: Domain Number of items Marginal Reliability (IRT) Active Hope Symptom Safe Social Total CRM The IRT analysis produced acceptable reliabilities, but further analysis showed something rather interesting:

11 Additional Information Provided by Rasch Rating Scale Model (IRT): Item Person Map High Score (High Recovery) Low Scores (Low Recovery) Consumers: Items: Q4 (3.36)-HOPE Q5 (2.57)- ACTIVE Q8 (2.14)- HOPE Q2 (1.72)- ACTIVE Q9 (1.29)- ACTIVE Q17 (1.16)- SOCIAL Q11 (.90)- SAFE Q7 (2.63)- HOPE Hard Items Q20 (1.76)- SYMP & Q21(2.08)- SYMP Q18 (1.43) SYMP Q14 (1.34)- SAFE & Q15 (1.33) SYMP Q12 (1.24)- SOCIAL Q19 (1.03) SOCIAL Q16 (.83)- SAFE Notice, that all of the items are at a higher level of recovery than the consumers This finding forced us to change our interpretation about the psychometrics of the scale Easy Items

12 Comparison of CTT and IRT results for Example 1 By only reviewing the CTT analysis the psychometrics seemed fine With the additional information provided by the Rasch model we understand that our questions are too difficult for our sample, Therefore, we need to create more items that will measure less recovery (to measure small changes)

13 Example 2: Recovery Markers Indicators usually associated with individual's recovery, but are not necessary for recovery (thus the term markers ) Includes 6 dimensions with varying response sets: Employment (8 response categories) Education/Training (7 response categories) Active/Growth orientation (6 response categories) Symptom Interference (5 response categories) Housing * (9 response categories) Engagement/role with service provider (6 response categories) Substance Abuse- level of use (6 response categories) Substance Abuse- level of change* (5 response categories)

14 Clinicians, rather than consumers complete this scale on a predetermined schedule dependent on the team type (High/medium CM versus outpatient)

15 Step 1: CTT Reliability and Factor Analysis Internal Reliability estimated Cronbach s alpha = 0.67 An EFA revealed 2 correlated factors (52.11%; 1 factor: 30%) A CFA was conducted on the 1 factor solution x²(11)= 26.32, p= (n = 776) RMSEA = GFI = 0.99 CFI = 0.99 This analysis told us that the scale could be improved, but did not explain how or where

16 Step 2: Partial Credit Rasch Model PERSONS MAP OF ITEMS <frequ> <less> 2 + T EDUCATION S. T EMPLOYMENT.#.##.### I #### ACTIVE/GROWTH SUBSTANCE_ABUSE-change 0.###### S+M.####### ENGAGEMENT/ROLE-ServProv.######### SUBSTANCE_ABUSE-use.######## SYMPTOM_INTERFERENCE.############ ########### M ########### S -1.###### +.#######.##### S.###.## HOUSING.## T. -2.# T <rare> <more> Index suggesting good model fit for persons -Mean Square Infit =.99 -Mean Square Outfit =1.0 Index suggesting moderate model fit for items -Mean Square Infit = Mean Square Outfit =1.03 Education & Employment are too difficult for the sample Housing is the easiest item Big gaps with no items measuring the majority of participants

17 Step 3: Nominal Response Model The Nominal Response model is based on the 2 PL and requires more participants (1,000 s) Allows us to view the order of the responses within an item, to make sure they are ordered We can change the response categories to make sure that they are ordered in terms of difficulty

18 Hard Example of Nominal Output Employment Education Active/Growth C- Active Job Search (3.29) F- Full time college (4.77) E- Part time college (.3.66) 4.0 F- Very high (3.41) D- Non-paid work/volunteer (2.64) Difficulty of Item H- Full time independent (1.59) G- Part time independent (.76) E- Part time supported (-.37) F- Full time supported (-1.22) B- Interest in work, no action (-1.99) A- No interest in work (-2.71) D- noncredit training (-.10) G- Recent Grad (-.20) C- Active education/training search (-1.00) B-interest in education, no action (-1.05) A- No interest in education (-2.82) E- High (-.75) D- Moderate out MH system (-1.62) B- Low (-2.28) C- Moderate in MH system (-2.37) Easy A- Very low (-3.88) There are issues of improper ordering, large gaps, where there are not responses & clumps of responses

19 We have met with our advisory committee and proposed some changes that will take care of the problems described Implementation is underway for the new version (2.0) and we should start collecting data soon

20 Review of Psychometric Process 1 st CTT analysis Determine reliability 2 nd Rasch modeling Determine model fit (reliability), participants ability level & item difficulty 3 rd Nominal Model Determine model fit (reliability) and ordering of responses within items As you increase the complexity of the measurement model, you also increase the assumptions required

21 Lessons Learned Requires time to educate yourself, critical to use appropriate model for your data In IRT literature be prepared to read conflicting pieces of information regarding model usage (Rasch vs. IRT) If you have stakeholders that want to be involved in the analysis (and we believe they should!), be prepared with example concepts (i.e. IQ) Sample size requirements Have resource to conduct analysis, stakeholder buy in Purchase software (winstep, bilog, etc..) Computer memory (Maximum Likelihood estimation)

22 Take home Message Measurement is a critical step in any evaluation IRT is beneficial and allows you to see more aspects of measurement than CTT alone As we increase our understanding with IRT, we also increase our assumptions Regardless of which method you use, understanding the benefits and limitations of your measurement model will help to interpret your data

23 Questions??? Contact Information: Kate DeRoche, M.A. (303) Antonio Olmos, Ph.D. (303)

24 IRT Resources IRT 101 Reise, S. P., Ainsworth, A. T. & Haviland, M. G. (2005). Item Response Theory: Fundamentals, Applications, and Promise in Psychological Research. Current Directions in Psychological Science, 14, IRT Books Hambleton, R. K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Newbury Park, CA: Sage Publication, Inc. Embretson, S. E. & Reise, S. P. (2000). Item Response Theory for Psychologists. Mahwah, NJ: Lawrence Erlbaum Associates And many more resources