Item Response Theory: Methods for the Analysis of Discrete Survey Response Data

Transcription

1 Item Response Theory: Methods for the Analysis of Discrete Survey Response Data ICPSR Summer Workshop at the University of Michigan June 29, 2015 July 3, 2015 Presented by: Dr. Jonathan Templin Department of Educational Psychology University of Kansas COURSE OVERVIEW Item Response Theory (IRT) is used in a number of disciplines including sociology, political science, psychology, human development, business, and communications, as well as in education where it began as a method for the analysis of educational tests. This course is geared to individuals who are interested in the foundations and applications of item response models as tools for measurement of latent traits across disciplines. The course is designed to acquaint students with the basics of the field of item response theory (IRT). To be successful in understanding analyses using IRT, two factors are important: (1) familiarity with various IRT models, and (2) the ability to interpret and apply these models appropriately. This course will begin with presentation of popular item response models, their estimation, and proper interpretation, and then continue reinforcing these lessons throughout the week with numerous examples and applications using data from different disciplines (including political science, education, and psychology). Additional topics will include test equating, test development with IRT, differential item functioning, and computerized adaptive testing. All topics will be taught in a manner which emphasizes a modern approach to IRT by comparing and contrasting IRT with other statistical methods that use latent variables or random effects (e.g., confirmatory factor analysis, diagnostic classification models). The course will be focused on accessibility, with technical detail presented only when necessary for responsible application of the methods and techniques discussed. Participants should be familiar with basic statistical models (e.g., ANOVA and regression) and basic psychometrics (e.g., classical test theory), but no prior experience with item response or other psychometric models is assumed. The course will utilize software developed for estimation of general latent variable models (Mplus). In addition to a course packet, you will have electronic access to all course materials, including overhead slides, analysis scripts, output files, relevant supporting documentation, and recommended readings.

2 TENTATIVE COURSE SCHEDULE* Day Time Topic Monday Foundations of Item Response Theory 9:00-10:15 Lecture 1: Historical Perspectives and Basic Statistical Prerequisites 10:30-11:45 Lecture 2: Basic IRT Concepts, Models, and Assumptions 1:15-2:30 Lecture 2, Continued 2:45-4:00 Lecture 3: Model Specifications and Scale Characteristics 4:00-5:00 Lab Activity 1: Introduction to Mplus Software for IRT Tuesday Estimation of IRT Models 9:00-10:15 Lecture 4: IRT Models for Polytomous Response Data 10:30-11:45 Lecture 5: Estimation of Item Response Models 1:15-2:30 Lecture 5, Continued 2:45-4:00 Lecture 6: Assessment of Model Fit 4:00-5:00 Lab Activity 2: Polytomous IRT Models Wednesday Reliability in IRT/Test Development/Computerized Adaptive Testing 9:00-10:15 Lecture 7: Latent Trait Reliability 10:30-11:45 Lecture 8: Test Development with IRT 1:15-2:30 Lecture 8, Continued 2:45-4:00 Lecture 9, Computerized Adaptive Testing 4:00-5:00 Lab Activity 3: Scale Development Thursday Friday Equating/Item and Test Bias/Differential Item Functioning 9:00-10:15 Lecture 10: Equating 10:30-11:45 Lecture 10, Continued 1:15-2:30 Lecture 11: Differential Item Functioning 2:45-4:00 Lecture 11, Continued 4:00-5:00 Lab Activity 4: Equating Advanced IRT Models and Topics 9:00-10:15 Lecture 12: Multidimensional IRT 10:30-11:45 Lecture 13: Diagnostic Classification Models 1:15-2:30 Lecture 13, Continued 2:45-4:00 Lecture 14: Conclusions: Comparing IRT with Other Models

3 GOOD GENERAL REFERENCES: References on Applications and Extensions of IRT de Ayala, R. J. (2009). The theory and practice of item response theory. New york: Guildford. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Psychology press. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory principles and applications. Boston, MA: Kluwer-Nijhoff Publishing. Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum. McDonald, R. P. (1999). Test theory: A unified approach. Mahwah, NJ: Lawrence Erlbaum. Thissen, D., & Wainer, H. (Eds.). (2001). Test Scoring. Mahwah, NJ: Lawrence Erlbaum. van der Linden, W. J., & Hambleton, R. K. (Eds.). (1997). Handbook of modern item responsetheory. New York, NY: Springer. PARAMETER ESTIMATION: Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. New York, NY: Marcel Dekker. Lord, F. M. (1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48, Lord, F. M. (1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal of Educational Measurement, 23, Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, Stone, C. A. (1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, BAYESIAN STATISTICS (GENERAL, NOT IRT-BASED): Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, D.B. (1995). Bayesian Data Analysis. London: Chapman & Hall.

4 TEST DEVELOPMENT: Donoghue, J. R. (1994). An empirical examination of the IRT information of polytomously scored reading items under the generalized partial credit model. Journal of Educational Measurement, 31, Green, D. R., Yen, W. M., & Burket, G. R. (1989). Experiences in the application of item response theory in test construction. Applied Measurement in Education, 2(4), POLYTOMOUS IRT MODELS: Andrich, D. (1978). Scaling attitude items constructed and scored in the Likert tradition. Educational and Psychological Measurement, 38, Koch, W. R. (1983). Likert scaling using the graded response model. Applied Psychological Measurement, 7, Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 60, Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monographs, No. 17. Sheehan, K., & Mislevy, R. (1990) Integrating cognitive and psychometric models to measure document literacy. Journal of Educational Measurement, 27, Thissen, D., & Steinberg, L. (1986) A taxonomy of item response models. Psychometrika, 31, ASSESSING MODEL-DATA FIT: DeAyala, R. J., & Hertzog, M. A. (1991). The assessment of dimensionality for use in item response theory. Multivariate Behavioral Research, 26, Drasgow, F., et al. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, TEST SCORE EQUATING: Baker, F. B. (1992). Equating tests under the graded response model. Applied Psychological Measurement, 16, Cohen, A. S., & Kim, S. H. (1998). An investigation of linking methods under the graded response model. Applied Psychological Measurement, 22,

5 Hanson, B.A. & Béguin, A. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices, 2nd edition. New York: Springer-Verlag. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, DIFFERENTIAL ITEM FUNCTIONING (DIF): Holland, P. W., & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum. Zwick, R., Thayer, D. T., & Mazzeo, J. (1997). Descriptive and inferential procedures for assessing differential item functioning in polytomous items. Applied Measurement in Education, 10, MULTIDIMENSIONAL IRT: Ackerman, T. A. (1996).Graphical representation of multidimensional item response theory analyses. Applied Psychological Measurement, 20, Ackerman, T. A. (1994).Using multidimensional item response theory to understand what items and tests are measuing. Applied Measurement in Education, 7, DeAyala, R. J. (1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18, Reckase, M. D. (1985, April). The difficulty of test items that measure more than one ability. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. DIAGNOSTIC CLASSIFICATION MODELS: Leighton, J., & Gierl, M. (Eds.). (2007). Cognitive diagnostic assessment in education: Theoryand applications. Cambridge University Press Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and Applications. New York: Guilford.