Exploring dimensionality of scores for mixedformat

Transcription

1 University of Iowa Iowa Research Online Theses and Dissertations Summer 2016 Exploring dimensionality of scores for mixedformat tests Mengyao Zhang University of Iowa Copyright 2016 Mengyao Zhang This dissertation is available at Iowa Research Online: Recommended Citation Zhang, Mengyao. "Exploring dimensionality of scores for mixed-format tests." PhD (Doctor of Philosophy) thesis, University of Iowa, Follow this and additional works at: Part of the Educational Psychology Commons

2 EXPLORING DIMENSIONALITY OF SCORES FOR MIXED-FORMAT TESTS by Mengyao Zhang A thesis submitted in partial fulfillment of the requirements for the Doctor of Philosophy degree in Psychological and Quantitative Foundations (Educational Measurement and Statistics) in the Graduate College of The University of Iowa August 2016 Thesis Supervisors: Professor Michael J. Kolen Associate Professor Won-Chan Lee!

3 Copyright by MENGYAO ZHANG 2016 All Rights Reserved!

4 Graduate College The University of Iowa Iowa City, Iowa CERTIFICATE OF APPROVAL PH.D. THESIS This is to certify that the Ph.D. thesis of Mengyao Zhang has been approved by the Examining Committee for the thesis requirement for the Doctor of Philosophy degree in Psychological and Quantitative Foundations (Educational Measurement and Statistics) at the August 2016 graduation. Thesis Committee: Michael J. Kolen, Thesis Supervisor Won-Chan Lee, Thesis Supervisor Robert L. Brennan Timothy N. Ansley Kate Cowles!

5 To Shilan, Li, Mingjie, and Aaron! ii!

6 ACKNOWLEDGEMENTS I would first like to thank Dr. Michael J. Kolen, my advisor and thesis supervisor, for seeing my potential and always encouraging me to pursue research that was of interest to me and of use to the profession. Without his great insights, invaluable feedback, constant support and patience throughout my study in Iowa, my dissertation and many other works would not have gone so smoothly. I would also like to thank Dr. Won-Chan Lee who served as one of the co-chairs of this dissertation and has continuously guided and encouraged me. He was always willing to discuss research ideas with me, and motivated me to see the big picture while paying attention to the details. I sincerely thank my other committee members: Dr. Robert Brennan, Dr. Timothy Ansley, and Dr. Kate Cowles. Their knowledge and advices greatly improved this dissertation. Dr. Robert Brennan shared many insightful comments concerning the issues dealt with in this dissertation, which greatly enriched this dissertation. I am also indebted to him for his support and encouragement at various stages of my study. A special thanks goes to Jane Persons whose edits and comments were invaluable. I also wish to thank the College Board for allowing me to use the Advanced Placement Exam data in my dissertation. Finally, I would like to thank the many friends and colleagues from whom I received help and support. Especially, Min Wang, our friendship means a lot to me. I am also truly thankful to my parents, Shilan and Li, for their support and understanding. To my husband, Mingjie, your unconditional love and support carries me through always. To my lovely son, Aaron, you are always my sunshine.! iii!

7 ABSTRACT Dimensionality assessment provides test developers and users with a better understanding of how test scores make human abilities concrete. Issues dealt with by dimensionality assessment include, but are not restricted to, (a) whether unidimensionality holds; (b) the number of dimensions influencing test scores; and (c) the relationships among items, among underlying dimensions, and between items and dimensions. Results from dimensionality assessment allow test developers and users to carefully validate specific interpretations and uses of test scores. The widespread use of mixed-format tests complicates dimensionality assessment both conceptually and methodologically. This dissertation is the first to propose a framework tailored for exploratory type of dimensionality assessment for mixed-format tests. Based on real data from three large-scale mixed-format tests, this dissertation examined the performance of a number of popular and promising dimensionality assessment methods and procedures. Major findings were summarized, along with more extensive descriptions of the similarities and dissimilarities among methods and across different test subject areas, forms, and sample sizes. Limitations and possible further research topics were also discussed.! iv!

8 PUBLIC ABSTRACT Dimensionality assessment provides test developers and users with a better understanding of how test scores make human abilities concrete. Issues dealt with by dimensionality assessment include, but are not restricted to, (a) whether unidimensionality holds; (b) the number of dimensions influencing test scores; and (c) the relationships among items, among underlying dimensions, and between items and dimensions. Results from dimensionality assessment allow test developers and users to carefully validate specific interpretations and uses of test scores. The widespread use of mixed-format tests complicates dimensionality assessment both conceptually and methodologically. This dissertation is the first to propose a framework tailored for exploratory type of dimensionality assessment for mixed-format tests. Based on real data from three large-scale mixed-format tests, this dissertation examined the performance of a number of popular and promising dimensionality assessment methods and procedures. Major findings were summarized, along with more extensive descriptions of the similarities and dissimilarities among methods and across different test subject areas, forms, and sample sizes. Limitations and possible further research topics were also discussed.! v!

9 TABLE OF CONTENTS LIST OF TABLES... viii LIST OF FIGURES... xi NOTATION... xiii CHAPTER I. INTRODUCTION...1 Defining Dimensionality...1 Related Concepts...2 Assessing Dimensionality...5 Exploratory versus Confirmatory Assessment...6 Operational Definitions of Dimensionality...6 Assessment Outcomes...8 Types of Statistical Procedure...9 Remark...10 Challenges Introduced by Mixed-Format Tests...10 Definitions of MC and CR Items...10 Potential Sources of Multidimensionality...11 Methodological Challenges in Dimensionality Assessment...12 Overview of Data Sources...14 Research Objectives and Questions...15 CHAPTER II. LITERATURE REVIEW...19 A Framework for Exploring Dimensionality of Mixed-Format Test Scores...19 Operational Definitions of Dimensionality...22 Defining Dimensionality in EFA...22 Defining Dimensionality in IRT...29 Description of Dimensionality Assessment Methods...34 Methods Using the EFA Definition of Dimensionality...34 Methods Using the IRT Defition of Dimensionality...39 Comparisons of Dimensionality Assessment Methods...44 Earlier Research...44 Hattie s and Tate s Comparative Studies...45 Review of Relevant Literature by Method...46 Item Format and Content Effects in Mixed-Format Tests...49 Item Format Effects...49 Item Content Effects...52 Summary...54 CHAPTER III. METHODOLOGY...56 Data Sources...56 Selection of Tests...56 Test Blueprints...57! vi!

10 Data Preparation...59 Data Description...60 Procedure...64 Methods Using the EFA Definition of Dimensionality...64 Methods Using the IRT Defition of Dimensionality...66 Evaluation Criteria...69 CHAPTER IV. RESULTS...77 Descriptive Statistics...77 Whether Unidimensionality Holds...80 Number of Dimensions...83 Dimensional Structure...89 Results of Item-Level EFA...90 Results of MIRT Cluster Analysis...97 Summary Item Format and Content Effects CHAPTER V. DISCUSSION Summary of Findings Research Question Research Question Research Question Research Question Research Question Research Question Limitations and Future Research Conclusions and Implications REFERENCES APPENDIX A. RESCORING CHEMISTRY TEST ITEMS APPENDIX B. SAMPLE MPLUS ITEM-LEVEL EFA SYNTAX APPENDIX C. SAMPLE FLEXMIRT MODEL ESTIMATION SYNTAX...196! vii!

11 LIST OF TABLES Table 3.1. Test Blueprints for the English Test, Form A and Form B...74 Table 3.2. Test Blueprints for the Spanish Test, Form A and Form B...74 Table 3.3. Test Blueprints for the Chemistry Test, Form A and Form B...74 Table 3.4. Item and Total Score Points for Selected Test Forms...74 Table 3.5. Dimensionality Assessment Methods Investigated...75 Table 4.1. Total Score Scale, Moments, and Reliability Coefficients Table 4.2. Disattenuated Correlations for the English Datasets Table 4.3. Disattenuated Correlations for the Spanish Datasets Table 4.4. Disattenuated Correlations for the Chemistry Datasets Table 4.5. Item Difficulty, Discrimination, and Irregular Items for MC Items Table 4.6. Item Difficulty, Discrimination, and Irregular Items for CR Items Table 4.7. Table 4.8. Table 4.9. Skewness of Frequency Distribution of Response Categories for CR Items Results of Kaiser s Rule, Percent Variance, Ratio of Eigenvalues, and EFA-Based Poly-DIMTEST Methods Claims of Unidimensionality Using Kaiser s Rule, Percent Variance, Ratio of Eigenvalues, and EFA-Based Poly-DIMTEST Methods Table Results of MAP and PA Methods Table Values of Model-Fit Statistics for the English Datasets Table Number of Items Substantially Related to Factors for the English Datasets Table Interfactor Correlations for the English Datasets Table Descriptive Statistics of Correlation Residuals for the English Datasets Table A 3-Factor Solution for the English Test, Baseline (Form A, N = 3,000) Table Item Clusters Produced by Item-Level EFA for the English Datasets...119! viii!

12 Table Values of Model-Fit Statistics for the Spanish Datasets Table Number of Items Substantially Related to Factors for the Spanish Datasets Table Interfactor Correlations for the Spanish Datasets Table Descriptive Statistics of Correlation Residuals for the Spanish Datasets Table A 4-Factor Solution for the Spanish Test, Baseline (Form A, N = 3,000) Table Item Clusters Produced by Item-Level EFA for the Spanish Datasets Table Values of Model-Fit Statistics for the Chemistry Datasets Table Number of Items Substantially Related to Factors for the Chemistry Datasets Table Interfactor Correlations for the Chemistry Datasets Table Descriptive Statistics of Correlation Residuals for the Chemistry Datasets Table Estimated Slope Parameters for the English Test, Baseline (Form A, N = 3,000) Table Stopping Rules and Suggested Number of Clusters for the English Datasets Table Item Clusters Produced by MIRT Cluster Analysis for the English Datasets Table Estimated Slope Parameters for the Spanish Test, Baseline (Form A, N = 3,000) Table Stopping Rules and Suggested Number of Clusters for the Spanish Datasets Table Item Clusters Produced by MIRT Cluster Analysis for the Spanish Datasets Table Estimated Slope Parameters for the Chemistry Test, Baseline (Form A, N = 3,000) Table Stopping Rules and Suggested Number of Clusters for the Chemistry Datasets...141! ix!

13 Table Item Clusters Produced by MIRT Cluster Analysis for the Chemistry Datasets Table Consistency in Dimensionality-Based Item Clusters between Item- Level EFA and MIRT Cluster Analysis Table Results of Format- and Content-Based Poly-DIMTEST Method, Baseline (Form A, N = 3,000) Table Dimensionality-Based Item Clusters Produced by Item-Level EFA and MIRT Cluster Analysis for the English Test, Baseline (Form A, N = 3,000) Table Dimensionality-Based Item Clusters Produced by Item-Level EFA and MIRT Cluster Analysis for the Spanish Test, Baseline (Form A, N = 3,000) Table A.1. Table A.2. Total Score Scale, Moments, and Reliability Coefficients for the Chemistry Datasets, Before and After Rescoring Disattenuated Correlations for the Chemistry Datasets, Before and After Rescoring...194! x!

14 LIST OF FIGURES Figure 2.1. An Example Framework for Exploring Dimensionality of Mixed- Format Test Scores...55 Figure 3.1. Proportions of Consistent and Inconsistent Structural Identifications...76 Figure 3.2. Example Proportions of Consistent and Inconsistent Structural Identifications...76 Figure 4.1. Summed MC Score by Proportion Correct for Two MC Items Figure 4.2. Scree Plots of the First 10 Eigenvalues of the Original and Smoothed Polychoric Correlation Matrices Figure 4.3. Scree Plots of the First 10 Eigenvalues of the Polychoric Correlation Matrices Figure 4.4. MAP Plots for the First 10 Principal Components, Baseline (Form A, N = 3,000) Figure 4.5. MAP Plots for the First 10 Principal Components, Comparisons Between Forms Figure 4.6. MAP Plots for the First 10 Principal Components, Comparisons Among Sample Sizes Figure 4.7. Plots of Eigenvalues Minus PA Threshold for the First 10 Principal Components, Baseline (Form A, N = 3,000) Figure 4.8. Plots of Eigenvalues Minus PA Threshold for the First 10 Principal Components, Comparisons Between Forms Figure 4.9. Plots of Eigenvalues Minus PA Threshold for the First 10 Principal Components, Comparisons Among Sample Sizes Figure R-Squares of 1-Factor Model and R-Square Improvements of 2-, 3-, 4-Factor Models for the English Datasets Figure R-Squares of 1-Factor Model and R-Square Improvements of 2-, 3-, 4-Factor Models for the Spanish Datasets Figure R-Squares of 1-Factor Model and R-Square Improvements of 2-, 3-, 4-Factor Models for the Chemistry Datasets Figure Cluster Dendrogram for the English Test, Baseline (Form A, N = 3,000)...161! xi!

15 Figure Cluster Dendrogram for the Spanish Test, Baseline (Form A, N = 3,000) Figure Cluster Dendrogram for the Chemistry Test, Baseline (Form A, N = 3,000)...163! xii!

16 NOTATION 1PL 2PL 3PL AP CH CINEG CFA CR DH EFA ES GR IRF IRT LS M3PL MAP MAP-R2 MAP-R4 MAP-P2 MAP-P4 MC MDIFF MDISC One-parameter logistic model in IRT Two-parameter logistic model in IRT Three-parameter logistic model in IRT College Board Advanced Placement Exams The Caliński and Harabasz (1974) index Common-item nonequivalent groups design in equating Confirmatory factor analysis Constructed-response item format The Duda and Hart (1973) index Exploratory factor analysis Common-item effect size The graded-response model in IRT Item response function in IRT Item response theory Least square estimation method Multidimensional extension of the 3PL model in IRT Minimum average partial procedure MAP with Pearson partial correlations squared MAP with Pearson partial correlations to the fourth power MAP with polychoric partial correlations squared MAP with polychoric partial correlations to the fourth power Multiple-choice item format Multidimensional item difficulty parameter in IRT Multidimensional item discrimination parameter in IRT! xiii!

17 MGR MIRT ML O1 O2 O3 O4 PA PA-MCm PA-MC95 PA-Pm PA-P95 PCA PD PFA RMSEA RMSR SEM SLI TOEFL Multidimensional extension of the GR model in IRT Multidimensional IRT Maximum likelihood estimation method Assessment outcome 1: whether unidimensionality holds Assessment outcome 2: the number of dimensions Assessment outcome 3: specific dimensional structure Assessment outcome 4: explanations of dimensions Parallel analysis PA based on Monte Carlo simulation with a mean threshold PA based on Monte Carlo simulation with a 95th-percentile threshold PA based on permutations with a mean threshold PA based on permutations with a 95th-percentile threshold Principal component analysis Positive definite matrix Principal factor analysis Root mean square error of approximation calculated by Mplus Root mean square residual calculated by Mplus Structural equation model Strong local independence Test of English as a Foreign Language TOEFL-iBT Internet-based version of TOEFL UIRT WLI Unidimensional IRT Weak local independence!! xiv!

18 ! Principal component loading matrix in PCA Factor loading (factor pattern) matrix in PFA!! Vector of item discriminations or slopes!! Item discrimination or slope parameter in IRT!! Item or category difficulty parameter in IRT!! Item pseudo-guessing parameter in IRT!! Unique factor loading matrix in PFA Matrix scaling a covariance matrix into a correlation matrix!! Number of common factors in PFA Number of latent traits in IRT Item or category intercept parameter in IRT!!! Number of essential dimensions in IRT!! Vector of common factors!! Identity matrix! and!! Items!! Number of items!!! Lower bound used in defining weak monotonicity in IRT!! Item response category!! Number of item response categories!! Number of items in AT1/AT2 when conducting Poly-DIMTEST Sample mean of common-item scores!! Number of examinees!! The polychoric correlation matrix of item scores!! Probability of a correct response in IRT!! Cumulative response function in IRT!! The Pearson correlation matrix of item scores!! Partial correlation matrix used when conducting the MAP! xv!

19 procedure!!! Reduced correlation matrix in PFA!! The Pearson correlation!! and!! Section score!!! Sample variance of common-item scores!! Stout s statistic used when conducting Poly-DIMTEST!! Thresholds used in defining the polychoric correlation!! Vector of unique factors in PFA!! Linear transformation matrix defining the combination of item scores!! Eigenvector of a matrix!! Vector of item scores!! Item score!! Vector of linear combinations of item scores!! Vector of principal components!! Linear combination of item scores!! Principal component!!!! Direction between the item vector and the first latent trait!!!!! Angular distance between two items!! Vector of latent traits!! Latent trait!! Variance-covariance matrix of principal components!! Eigenvalue of a matrix!! Latent variable used in defining the polychoric correlation!!!!!!! Feldt s reliability coefficient estimate! xvi!

20 !!!!!! Disattenuated correlation between two section scores!! Matrix of correlations between common factors in PFA!! Consistency index used to compare dimensional solutions from using two methods!!!! Proportion of item pairs assigned to different dimensions by two methods!!"! Proportion of item pairs assigned to different dimensions by the first method but assigned to the same dimension by the second method!!!! Proportion of item pairs assigned to the same dimension by two methods!!"! Proportion of item pairs assigned to the same dimension by the first method but assigned to different dimensions by the second method! xvii!

21 1 CHAPTER I INTRODUCTION An effective educational test reflects important differences among examinees on one or more dimensions it intends to measure (Yen & Fitzpatrick, 2006). Depending on the specific purposes and uses of a test, such dimensions could be reading comprehension, mathematical reasoning, or chemistry laboratory skills, to name a few. Dimensionality assessment provides empirical knowledge of how these dimensions are reflected by the actual test data and whether or not some unexpected dimensions emerge. The dimensional structure of the data is one of the critical pieces of evidence used to validate interpretations and uses of test scores. Defining Dimensionality Examination of dimensionality is more challenging than it might seem. Perhaps the most fundamental problem facing researchers and practitioners is: what is meant by dimensionality? Hattie (1981, 1984, 1985) first distinguished dimensionality from reliability, homogeneity, and internal consistency. He defined dimensionality in a straightforward way that a unidimensional test is one that has one latent trait underlying the data (Hattie, 1985, p. 157). Following his rationale, a multidimensional test has two or more distinct latent traits underlying the data. However, since latent trait itself is another abstract and ambiguous concept, such definitions are more theoretical than practical and as such shed little light on how dimensionality could be evaluated. As a result, various operational definitions of dimensionality have been developed, serving as the basis for different dimensionality assessment methods. Among these operational definitions, unfortunately, no consensus exists on which one best describes the nature of dimensionality. Researchers and practitioners just implicitly decide on the lens through which they view dimensionality as they choose one or more methods to analyze their own data.

22 2 The term dimensionality has also been used somewhat loosely in the existing literature, appearing at times as a feature of a test and at others as a feature of test scores (Reckase, 2009). Even when the authors expressed literally that a test is unidimensional (or multidimensional), they might imply that dimensionality is some kind of underlying characteristics of the given data (e.g., see Hattie s definition of unidimensionality: a unidimensional test is one that has one latent trait underlying the data). This dissertation adopted the following viewpoint that dimensionality is a characteristic of test scores, rather than that of the test or test form. More specifically, dimensionality represents the dynamic interaction between a test and an examinee population who respond to the test items (Ackerman, 1994; Reckase, 2009; Tate, 2002). In other words, dimensionality is population dependent by definition, although the dimensional structure of test scores does not necessarily change when different examinee populations are involved. Related Concepts Two pairs of contrasting concepts have been mentioned frequently in previous research and application studies on dimensionality assessment, though the names of the specific concept may vary: (a) strict versus essential dimensionality (e.g., McDonald, 1981; Nandakumar, 1991; Stout, 1987, 1990; Tate, 2002, 2003), and (b) fixed versus random dimensionality (e.g., Wainer & Thissen, 1996). In the first pair of concepts, mathematical definitions of strict and essential dimensionality have been proposed using item response theory (IRT) (Stout, 1987), which are explained in Chapter II. Roughly stated here, the former considers all possible dimensions reflected by the data, whereas the latter is only concerned with a limited number of major or dominant dimensions. For operational mixed-format tests, perfect unidimensionality, where all items strictly measure a single dimension, likely rarely occurs. Therefore, this dissertation evaluates essential dimensionality of data and sometimes uses the two terms dimensionality and essential dimensionality

23 3 interchangeably without explicit distinction being made. Some questions of interest could be: Are the data (essentially) unidimensional? How many (major or dominant) dimensions are reflected in the data? It should be noted that essential dimensionality of data is examined in different ways across methods, some of which rely on strong theoretical foundations, whereas others rely greatly on subjective judgments. In the second pair of concepts, an essential distinction between fixed and random dimensionality is the intention of test developers (Wainer & Thissen, 1996). In particular, when test developers build a test to cover a broad range of content subdomains or cognitive skills, they might expect to see some degree of multidimensionality in the scores. On the other hand, after the test is administered and responses are scored, test developers might encounter some unexpected dimensions, for example, due to inappropriate reading load or speededness. Wainer and Thissen (1996) named these concepts fixed and random multidimensionality, respectively. However, it is sometimes difficult to determine whether or not dimensionality is as expected. Take item format as an example. When test developers decide to combine multiple-choice (MC) items with constructed-response (CR) items to create a test form, they may or may not expect to find multidimensionality due to the item format, depending on their conceptual framework regarding whether or not different traits are related to MC and CR items. Furthermore, as will be discussed in greater detail in the literature review in Chapter II, data from a test that consists of different item formats or content subdomains may not show clear multidimensionality at all. Nevertheless, this pair of concepts conveys an important aspect of the essence of dimensionality: it is not solely about statistics. Final decisions on the dimensionality of test scores are generally made based on both statistical results and substantive judgments (Ackerman, Gierl, & Walker, 2003). Especially as unidimensionality appears untenable,

24 4 efforts are needed to find plausible interpretations of dimensions, which could be associated with aspects of test development (the intention of test developers), administration, scoring, and other related policies and practices. Dimensionality assessment is as much an art as it is a science. Dimensionality is a characteristic of test scores, and is thereby affected by the specific scoring method used to produce those scores. For example, a special issue in dimensionality assessment for mixed-format tests stems from the different types of item scores used with MC and CR items, which is discussed later in this chapter. Even for an MC-only test, either number-correct or various correction-for-guessing scoring methods could be considered to score item responses. The choice of scoring method often affects how different dimensionality assessment methods behave and what results they provide. Thus, it becomes necessary to describe how scores are calculated when reporting specific dimensional solutions. When the scoring method is changed, it is prudent to replicate dimensionality assessment so as to determine whether or not different scoring methods lead to substantially different conclusions on dimensional structure. Score reliability is interrelated with dimensionality. Reliability refers to consistency of scores of specified examinees over replicate measurements (Feldt & Brennan, 1989; Haertel, 2006). Consider the situation in which scores are not sufficiently reliable. In this situation, scores of examinees that result from repeated measurements vary to a large extent. Consequently, any dimensional solution based on scores from one replication appears less convincing. Dimensionality in turn affects the evaluation of reliability. Compared with unidimensional data, if the dimensional structure of the data is complex, creating replications of measurements of the same quality demands considerably more effort. Selecting a proper way to express reliability (e.g., which type of reliability coefficients to calculate) also needs careful consideration. Dimensionality also shares a close relationship with equating, a statistical process that enables comparability of scores on multiple forms of a test (Kolen & Brennan, 2014).

25 5 Dimensionality assessment could help inform various decisions for equating, such as construction of test forms and selection of equating models and procedures. One example is how to build a common-item set for equating of mixed-format test forms under a common-item nonequivalent groups (CINEG; Kolen & Brennan, 2014) design. In general, the common items would better represent the content and statistical characteristics of the total test, so that performance on the common items could be generalized to that on the rest of the test. But for mixed-format tests that contain both MC and CR items, is there an additional need for balancing the item format of the common items? Previous research has revealed that the composition of the common-item set, either MC-only or format-representative, could lead to acceptable or unacceptable equating results under certain conditions (Kolen & Lee, 2014). Dimensionality is an important factor to consider, because no such concern might arise when there is no substantial multidimensionality due to the item format. Another example is whether to apply unidimensional IRT (UIRT) or multidimensional IRT (MIRT) models for equating. As their names suggest, UIRT models assume unidimensionality of data (or essential unidimensionality of data in most practical situations), whereas MIRT models are capable of handling multidimensionality. Evaluation of the unidimensionality assumption is a prerequisite for using UIRT models. Assessment of the degree of multidimensionality and particular dimensional structure helps in the decision about whether the benefit of using a MIRT model exceeds the price paid (e.g., heavy computation) and what MIRT models would be appropriate. Assessing Dimensionality This section introduces a rough classification of dimensionality assessment and overviews three major aspects in dimensionality assessment, including operational definition, assessment outcome, and type of statistical procedure. These aspects later

26 6 serve as components for developing a framework to classify and discuss a variety of dimensionality assessment methods, which will be delineated in Chapter II. Exploratory versus Confirmatory Assessment Dimensionality could be evaluated in both exploratory and confirmatory manners (Reckase, 2009; Svetina & Levy, 2014). Exploratory dimensionality assessment, which is the focus of this dissertation, is typically employed when there is no clear hypothesis or evidence concerning the dimensional structure of the given data. It has been routinely conducted, either alone or combined with confirmatory dimensionality assessment, in operational testing programs to check the alignment of actual dimensionality with the intended dimensionality (e.g., Fu, Chung, & Wise, 2013; Jang & Roussos, 2007; Wilson, 2000; Zwick, 1987). Exploratory dimensionality assessment also often serves as an integrated part of a preliminary analysis before studying other psychometric procedures (e.g., MIRT equating, see Brossman & Lee, 2013). However, the borderline between the two types of dimensionality assessment is blurry, and any given method is more likely to lie somewhere on the exploratory-confirmatory spectrum (Svetina & Levy, 2014). The dimensionality assessment methods that are more exploratory in nature are investigated in this dissertation, although the findings from these methods might also be useful for some confirmatory scenarios. Operational Definitions of Dimensionality As noted previously, there is no generally agreed upon unified operational definition of dimensionality. Discrepancies might occur among results from methods driven by different perspectives towards dimensionality. Two perspectives are considered in this dissertation, including an exploratory factor analysis (EFA) perspective and an IRT perspective. A more in-depth description of the two perspectives will be provided in Chapter II.

27 7 Dimensionality has long been analyzed using factor analysis. For example, see Hattie (1985), Reckase (2009), Stone and Yeh (2006), and Velicer, Eaton, and Fava (2000), for overviews of the application of factor analysis methods to dimensionality assessment. EFA, in a broad sense, stands for a class of statistical procedures used to explain the observed variances and covariances (Kline, 2010). Unlike confirmatory factor analysis (CFA), EFA does not require the use of a hypothesized dimensional structure, which seems advantageous for exploratory purposes. From the perspective of EFA, the number of dimensions equals the number of components or factors to retain. The data are considered to be unidimensional when only one component or factor is kept, and otherwise some degree of multidimensionality emerges. Dimensional structure underlying the data corresponds to a particular factor solution produced by EFA. Alternatively, dimensionality could be defined in the context of IRT. Advocates of this perspective believe that IRT allows for a clear and precise way to understand the concepts of unidimensionality and multidimensionality (Hattie, 1985; Nandakumar, 1991; Stout, 1987; Stout et al., 1996; Zhang & Stout, 1999a, 1999b). A traditional IRT definition of dimensionality is the minimum number of latent traits required for a locally independent and monotone model, which was presented in Stout (1990) and later elaborated in Nandakumar (1991), Stout et al. (1996), and Zhang and Stout (1999a, 1999b). Specifically, local independence implies that, after controlling for underlying latent traits, item responses are either mutually independent or pairwise uncorrelated, depending on whether strong local independence (SLI; Lord, 1980) or weak local independence (WLI; McDonald, 1981) is considered. Monotonicity indicates that the probability of correctly answering an item changes monotonically with values of latent traits. When a single latent trait is sufficient to produce such a model, the data are considered to be unidimensional. If the data are not unidimensional, the number of latent traits required defines the number of dimensions. Based on IRT, Stout (1987) also defined essential dimensionality as the number of major or dominant latent traits, which

28 8 turns out to be one of the most substantial concepts in designing some nonparametric dimensionality assessment procedures. Similar ideas were conveyed in some of the factor analysis literature (for a brief overview, see Stout, 1990), but a rigorous mathematical definition of essential dimensionality was first provided in Stout (1987) using IRT. Assessment Outcomes Dimensionality assessment, in general, is intended to assist researchers and practitioners in gaining a more accurate understanding of a test s internal structure. In earlier research, the primary purpose of dimensionality assessment was to check whether unidimensionality holds, because it was considered as one of the most desirable characteristics (Hattie, 1984, 1985). As more and more complex educational tests are developed, issues of multidimensionality have arisen more frequently; increasingly, dimensionality assessment has been applied to investigate multidimensional structure when unidimensionality looks doubtful (Svetina & Levy, 2014; Tate, 2003; Zhang & Stout, 1999a). Furr and Bacharach (2003) characterized the purposes of general dimensionality assessment to (a) decide on the number of dimensions, (b) estimate the correlation between multiple dimensions (if any), and (c) map statistical dimensions to psychological attributes. Taking into account different perspectives of previous studies (e.g., Furr & Bacharach, 2003; Hattie, 1985; Svetina & Levy, 2014; Tate, 2003; Zhang & Stout, 1999a) and personal experiences in analyzing dimensionality of operational mixedformat test data, this dissertation addresses the following major outcomes for exploratory dimensionality assessment: O1. Whether unidimensionality holds; O2. The number of dimensions; and O3. Specific dimensional structure (i.e., how items cluster around multiple underlying dimensions).

29 9 Information and documentation collected in addition to the test scores might also facilitate one further outcome (O4) including possible and meaningful explanations of emerging dimensions. Ackerman et al. (2003) among other researchers and practitioners have discussed some useful practices for making judgments on how and why a certain dimensional pattern is reflected by the data. For example, they suggested practitioners at least consider test specifications and conduct some additional content and psychological analyses to inform the dimensionality assessment. Because the primary purpose of this dissertation is to compare different dimensionality assessment methods using real data, attention was mostly paid to the first three outcomes, O1, O2, and O3; a brief discussion related to O4 is included but not emphasized. Types of Statistical Procedure Depending on the statistical procedure used, dimensionality assessment methods could be viewed as members from two different families, parametric and nonparametric. Parametric methods rely on certain psychometric and statistical models and assumptions, whereas nonparametric methods are more data-oriented and require few or weak models and assumptions. It is misleading to establish hasty connections between type of procedure (i.e., parametric or nonparametric) and definition of dimensionality (i.e., EFA or IRT) while ignoring the essential difference between parametric and nonparametric methods as indicated above. In particular, not all EFA-based methods are parametric, nor are all IRTbased methods nonparametric. For EFA, an example is Buja and Eyuboglu s (1992) version of parallel analysis (PA). Unlike the original PA (Horn, 1965) that is a parametric procedure, their version possesses nonparametric properties by replacing the normality assumption with the permutation principle. In IRT, although some nonparametric methods based on conditional covariances have been widely used, such as DIMTEST

30 10 (Nandakumar & Stout, 1993;; Stout, 1987), Miller and Hirsch s (1992) method appears to be parametric as it greatly depends on MIRT models. Remarks Variations in all of these aspects directly and indirectly impact the performance of dimensionality assessment methods. Different methods often yield dissimilar dimensional solutions, but no method has been clearly demonstrated to be universally satisfactory (Hakstian & Muller, 1973; Hambleton & Rovinelli, 1986; Hattie, 1984; Mroch & Bolt, 2006; Nandakumar, 1994; Stone & Yeh, 2006; Svetina & Levy, 2014; Tate, 2003; van Abswoude, van der Ark, & Sijtsma, 2004). Ease of implementation, and especially software accessibility, also plays a subtle role in dimensionality assessment. Some seemingly promising methods are not accompanied by any widely available software, which could compromise their practical applications. In contrast, default methods of some popular software packages might distort understanding of dimensionality to a considerable extent. Researchers and practitioners have to make several decisions to weigh potential gains and losses when selecting specific methods. Challenges Introduced by Mixed-Format Tests The goal of this dissertation is to understand performance of some popular and promising dimensionality assessment methods in mixed-format tests containing MC and CR items. In this section, definitions of MC and CR items are provided, followed by discussions on potential sources of multidimensionality and challenges of dimensionality assessment in the context of mixed-format tests. Definitions of MC and CR Items An MC item requires examinees to select one correct answer from a list of possible answers (Downing, 2006). Typically, an MC item is composed of a stem and four or five options, including one best answer and several distractors. In order to provide more context for an item, a stimulus such as a passage, table, or graph might also be

31 11 presented. Such a stimulus is either linked to a single item or shared by a set of items known as the testlet (Wainer & Kiely, 1987; Wainer & Lewis, 1990). Although different scoring methods could be used for scoring MC items, the number-correct scoring method is considered in this dissertation. Specifically, responses are dichotomously scored as 0 for an incorrect answer and 1 for a correct answer without penalty for guessing. A CR item, in contrast, requires examinees to construct their own answers. Various types of CR items have been seen in operational tests, such as short answer, essay, and speaking prompts. CR items are typically scored on an integer scale, such as , according to some predetermined scoring rubrics. Human raters, machine raters, or both could be involved in the scoring for CR items; for tests applying multiple raters, the number of raters and methods coordinating ratings vary across different tests. Advantages and limitations of MC and CR items have been discussed extensively in the literature (for a review, see Schmeiser & Welch, 2006). To combine strengths of different item formats and mitigate possible weaknesses of using either MC or CR items alone, many state and national educational testing programs have adopted mixed-format tests. The widespread use of mixed-format tests, however, complicates dimensionality assessment both conceptually and methodologically, as discussed next. Potential Sources of Multidimensionality Conceptually, data from mixed-format tests tend to have a more complex dimensional structure compared to those from single-format tests. A unique source of multidimensionality associated with mixed-format tests is the item format. However, previous research studies on whether varying item format introduces extra dimensions known as the item format effect (e.g., Bennett, Rock, & Wang, 1991; Bridgeman & Rock, 1993; Hohensin & Kubinger, 2011; Lissitz, Hou, & Slater, 2012; Manhart, 1996; Perkhounkova & Dunbar, 1999; Thissen, Wainer, & Wang, 1994; Traub, 1993; Traub & Fisher, 1977) have shown mixed findings.

32 12 Data from mixed-format tests are also vulnerable to the same potential sources of multidimensionality that may affect single-format tests. One major source is the item content. As indicated in Wainer and Thissen (1996), when a test is developed to measure several content subdomains, there is a natural view that some degree of multidimensionality might occur. However, such a view has not always been supported by empirical studies on single-format and mixed-format educational tests (e.g., Perkhounkova & Dunbar, 1999; Wilson, 2000; Zwick, 1987). Further, for some mixedformat tests, item content and format entangle themselves to influence the dimensional structure of the data, making dimensionality assessment even more difficult (Perkhounkova & Dunbar, 1999). In addition to item format and content, there are a number of factors that may cause the presence of multidimensionality in data from mixed-format tests. For example, Yen (1993) included passage or stimulus dependence in the testlet, speededness, rater effects, and inappropriate external interference among potential causes of local dependence and, given the close relationship between dimensionality and local dependence from the IRT perspective, those reasons likely lead to multidimensional data. This dissertation, however, focuses on two of the major sources: item format and content. Methodological Challenges in Dimensionality Assessment A methodological challenge facing dimensionality assessment of mixed-format test data stems from different types of item scores used with MC and CR items. Typically MC items are dichotomously scored, whereas CR items are polytomously scored. Thus, an ideal method for analyzing mixed-format test data should be able to handle dichotomous and polytomous item data simultaneously, which eliminates some widely used procedures developed for MC-only tests (Svetina & Levy, 2012). Those methods might still be used after polytomous data are dichotomized by collapsing multiple

33 13 categories into two categories (Svetina & Levy, 2014). However, unique information provided by CR items is lost during the dichotomization process. Even for those methods that appear technically feasible for mixed-format tests, their performance in this new context has not been studied as fully as would be desired. Fundamentally, there is a need to devise a general framework to categorize a variety of methods currently available. Some rough classifications have been suggested over the past few decades. For instance, methods have been classified according to their theoretical underpinnings (Hattie, 1985), types of procedures (Tate, 2003; Zhang & Stout, 1999a), or levels of complexity in assessment (Zhang & Stout, 1999a). Svetina and Levy (2012, 2014) put forward frameworks for dimensionality assessment methods and software packages with more sophisticated structure. But most of these frameworks were proposed based on dichotomous data (or requiring the dichotomization process prior to dimensionality assessment), so that their conclusions are not directly applicable to the context of mixed-format tests. To the best of the author s knowledge, there is currently no framework specifically for mixed-format tests that has been built with a clear focus on exploratory dimensionality assessment. There is also a need for more empirical comparisons of methods. Whereas a number of comparative studies have been done for MC-only tests (e.g., Hattie, 1984; Mroch & Bolt, 2006; Nandakumar, 1994; Stone & Yeh, 2006; Tate, 2003; van Abswoude et al., 2004; Svetina & Levy, 2014; Velicer et al., 2000), few studies have been done based on data from operational mixed-format tests. One of the consequences of this is a lack of recommendations and guidelines for researchers and practitioners who want to select methods that would be more beneficial for their own mixed-format test data, needs, and resources.

34 14 Overview of Data Sources An overview of the College Board Advanced Placement (AP) Exams, which are examples of mixed-format tests, is provided in this section. Data used in this dissertation were from several AP Exams. The AP Exam is a critical component of the AP Program that offers high school students the opportunity to receive college-level course credit and/or advanced placement in many colleges and universities in the United States (The College Board, 2014). There are over thirty AP Exams assessing students proficiency in subjects from six general areas: Arts, English, History & Social Science, Math & Computer Science, Sciences, and World Languages & Cultures. Information and documents concerning the courses and exams are available on the College Board website ( A typical AP Exam is comprised of both the MC and CR sections and covers multiple content subdomains or cognitive skills aligning to the proposed test specifications (The College Board, 2014). Yet the test length, test configuration (e.g., content and format distribution), and testing time can greatly vary depending on the examination. In the MC section, MC items may be stand-alone or grouped into testlets sharing common passages, audio scripts, tables, problem-solving settings, or other types of stimuli. Typically, MC items have four or five options each. Starting from the 2011 administration, number-correct scoring has been used for MC items, meaning that only correct answers count and no penalty is imposed for wrong answers (Kolen & Lee, 2011). In the CR section, various types of CR items are employed to best fulfill the proposed test purposes. Some typical types of CR items include short answer, long answer, synthesis essay, and speaking prompts under different scenarios. CR items are scored by human raters following standardized scoring rubrics. There is considerable diversity in the number of response categories and the resulting score range of the CR item among tests and items.

35 15 Operationally, a composite score is calculated based on weighted MC and CR section scores. In order to obtain interchangeable scores from different forms of the test, equating is conducted under the CINEG design (Kolen & Lee, 2011). A set of MC items resembling the total test in terms of content and statistical characteristics is treated as the common-item set for operational equating. The composite score is later converted into the AP Grade for each examinee. The AP Grades of 1 to 5 inform how well an examinee is qualified to receive college-level credit in a specific subject area (The College Board, 2014). Three AP Exams are used as illustrative examples in this dissertation: AP English Language and Composition (English), AP Spanish Language and Culture (Spanish), and AP Chemistry (Chemistry). It is important to note that a sequence of modifications of operational data were made in order to better address the research objectives and questions of this dissertation. As a result, the characteristics of these tests and samples are no longer similar to those in the operational settings, and results and findings in the empirical study have no direct implication for operational AP Exams. The primary focus of this dissertation is on the performance of different exploratory dimensionality assessment methods under varying conditions, and not on the characteristics of operational AP Exams. Research Objectives and Questions The intent of this dissertation is to achieve two main research objectives. First, a framework is built for considering various dimensionality assessment methods. Second, empirical method comparisons are conducted to investigate how different methods behave relative to each other under realistic conditions. Both real and simulated data are useful for the method comparison, but the former is of particular interest in this dissertation. The use of real data is important for several reasons. First, without a general knowledge of how specific dimensionality assessment

36 16 methods perform for operational mixed-format tests, it is difficult to design simulations. Unfortunately, such knowledge is scarce in the literature. It is real data analysis that allows for patterns of similarities and dissimilarities among methods to be observed and understood. Second, the inevitable biasing influence of the data generation method on the findings when using simulated data vanishes when real data are used. With real data studies, no dimensionality assessment method would be unfairly advantaged or disadvantaged due to the way the artificial multidimensional data were created. In spite of the fact that empirical studies alone cannot lead to definitive conclusions about which method is the best to use, the findings can still be valuable for researchers and practitioners when selecting among different methods. Not all possible methods are examined in this dissertation, but the most popular and promising methods for exploring dimensionality of mixed-format test data are included. Specific methods include Kaiser s rule (Kaiser, 1960), percent variance (Reckase, 1979), ratio of eigenvalues (Hattie, 1985; Lord, 1980), scree test (Cattell, 1966), original and refined versions of minimum average partial (MAP) procedure (Velicer, 1976; Velicer et al., 2000), original and refined versions of PA (Buja & Eyuboglu, 1992; Glorfeld, 1995; Horn, 1965), item-level EFA (Muthén & Muthén, ), exploratory and confirmatory modes of Poly-DIMTEST (Nandakumar, Yu, Li, & Stout, 1998), and MIRT cluster analysis (Miller & Hirsch, 1992). These methods will be described in Chapter II of this dissertation. Given the large number of methods available for examining dimensionality of mixed-format test data, the intent of the empirical study is to evaluate the consistency of their results. That is, the extent to which these methods provide similar dimensional solutions. Considering possible assessment outcomes, results are compared with regard to (O1) whether unidimensionality holds, (O2) the number of dimensions, and (O3) specific dimensional structure. Based on previous work on MC-only tests, discrepancies are likely