Introduction to Measurement

Transcription

1 This is a chapter excerpt from Guilford Publications. The Theory and Practice of Item Response Theory, by R. J. de Ayala. Copyright Introduction to Measurement I often say that when you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the state of science, whatever the matter may be. Sir William Thomson (Lord Kelvin) (1891, p. 80) This book is about a particular measurement perspective called item response theory (IRT), latent trait theory, or item characteristic curve theory. To understand this measurement perspective, we need to address what we mean by the concept of measurement. Measurement can be defined in many different ways. A classic definition is that measurement is the assignment of numerals to objects or events according to rules. The fact that numerals can be assigned under different rules leads to different kinds of scales and different kinds of measurement (Stevens, 1946, p. 677). Although commonly used in introductory measurement and statistics texts, this definition reflects a rather limited view. Measurement is more than just the assignment of numbers according to rules (i.e., labeling); it is a process by which an attempt is made to understand the nature of a variable (cf. Bridgman, 1928). Moreover, whether the process results in numeric values with inherent properties or the identification of different classes depends on whether we conceptualize the variable of interest as continuous or categorical. IRT provides one particular mathematical technique for performing measurement in which the variable is considered to be continuous in nature. Measurement For a simple example of measurement as a process, imagine that a researcher is interested in measuring generalized anxiety. Anxiety may be loosely defined as feelings that may range from general uneasiness to incapacitating attacks of terror. Because the very nature of 1

2 2 THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY anxiety involves feelings, it is not possible to directly observe anxiety. As such, anxiety is an unobservable or latent variable or construct. The measurement process involves deciding whether our latent variable, anxiety, should be conceptualized as categorical, continuous, or both. In the categorical case we would classify individuals into qualitatively different latent groups so that, for example, one group may be interpreted as representing individuals with incapacitating anxiety and another group representing individuals without anxiety. In this conceptualization the persons differ from one another in kind on the latent variable. Typically, these latent categories are referred to as latent classes. Alternatively, anxiety could be conceptualized as continuous. From this perspective, individuals differ from one another in their quantity of the latent variable. Thus, we might label the ends of the latent continuum as, say, high anxiety and low anxiety. When the latent variable is conceptualized as having categorical and continuous facets, then we have a combination of one or more latent classes and one or more latent continua. In this case, the latent classes are subpopulations that are homogeneous with respect to the variable of interest, but differ from one another in kind. Within each of these classes there is a latent continuum on which the individuals within the class may be located. For example, assume that our sample of respondents consists of two classes. One class could consist of individuals whose anxiety is so severe that they suffer from incapacitating attacks of terror. As such, these individuals are so qualitatively different from other persons that they need to be addressed separately from those whose anxiety is not so severe. Therefore, the second class contains individuals who do not suffer from incapacitating attacks of terror. Within each of these classes we have a latent continuum on which we locate the class s respondents. Although we cannot observe our latent variable, its existence may be inferred from behavioral manifestations or manifest variables (e.g., restlessness, sleeping difficulties, headaches, trembling, muscle tension, item responses, self-reports). These manifestations allow for several different approaches to measuring generalized anxiety. For example, one approach may involve physiological assessment via an electromyogram of the degree of muscle tension. Other approaches might involve recording the number of hours spent sleeping or the frequency and duration of headaches, using a galvanic skin response (GSR) feedback device to assess sweat gland activity, or more psychological approaches, such as asking a series of questions. These approaches, either individually or collectively, provide our operational definition of generalized anxiety (Bridgman, 1928). That is, our operational definition specifies how we go about collecting our observations (i.e., the latent variable s manifestations). Stated concisely, our interest is in our latent variable and its operational definition is a means to that end. The measurement process, so far, has involved our conceptualization of the latent variable s nature and its operational definition. We also need to decide on the correspondence between our observations of the individuals anxiety levels and their locations on the continuum and/or in a class. In general, scaling is the process of establishing the correspondence between the observation data and the persons locations on the latent variable. Once we have our individuals located on the latent variable, we can then compare them to one another. IRT is one approach to establishing this correspondence between the observation data and the persons locations on the latent variable. Examples of other relevant scaling processes are Guttman Scalogram analysis (Guttman, 1950), Coombs Unfolding (Coombs,

3 Introduction to Measurement ), and the various Thurstone approaches (Thurstone, 1925, 1928, 1938). Alternative scaling approaches may be found in Dunn-Rankin, Knezek, Wallace, and Zhang (2004), Gulliksen (1987), Maranell (1974), and Nunnally and Bernstein (1994). Some Measurement Issues Before proceeding to discuss various latent variable methods for scaling our observations, we need to discuss four issues. The first issue involves the consistency of the measures. By way of analogy, assume that we are measuring the length of a box. If our repeated measurements of the length of the box were constant, then these measurements would be considered to be highly consistent or to have high reliability. However, if these repeated measurements varied wildly from one another, then they would be considered to have low consistency or to have low reliability. In the former case, our measurements would have a small amount of error, whereas in the latter they would have a comparatively larger amount of error. The consistency (or lack thereof) would affect our confidence in the measurements. That is, in the first scenario we would have greater confidence in our measurements than in the second scenario. The second issue concerns the validity of the measures. Although there are various types of validity, we define validity as the degree to which our measures are actually manifestations of the latent variable. As a contrarian example, assume we use the frequency and duration of headaches approach for measuring anxiety. Although some persons may recognize that there might be a relationship between frequency and duration of headaches and anxiety level, they may not consider this approach, in and of itself, to be an accurate representation of anxiety. In short, simply because we make a measure does not mean that the measure necessarily results in an accurate reflection of the variable of theoretical interest (i.e., our measurements may or may not have validity). A necessary, but not sufficient condition for our measurements to have validity is that they possess a high degree of reliability. Therefore, it is necessary to be concerned not only with the consistency of our measurements, but also with their validity. Obtaining validity evidence is part of the measurement process. The third issue concerns a desirable property we would like our measurements to possess. Thurstone (1928) noted that a measuring instrument must not be seriously affected in its measuring function by the object of measurement. In other words, we would like our measurement instrument to be independent of what it is we are measuring. If this is true, then the instrument possesses the property of invariance. For instance, if we measure the size of a shoe box by using a meter stick, then the measurement instrument (i.e., the meter stick) is not affected by and is independent of which box is measured. Contrast this with the situation in which measuring a shoe box s size is done not by using a meter stick, but by stretching a string along the shortest dimension of the box and cutting the string so that its length equals the shortest dimension. This string would serve as our measurement instrument and we would use it to measure the other two dimensions of the box. In short, the measurements would be multiples of the shortest dimension. Then suppose we use this approach to measure a cereal box. That is, for the cereal box its shortest dimension is used

4 4 THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY to define the measurement instrument. Obviously, the box we are measuring affects our measurement instrument and our measurements would not possess the invariance property. Without invariance our comparisons across different boxes would have limited utility. The final issue we present brings us back to the classic definition of measurement mentioned above. Depending on which approach we use to measure anxiety (i.e., GSR, duration of headache, item responses, etc.), the measurements have certain inherent properties that affect how we interpret their information. For instance, the duration of headaches approach produces measurements that cannot be negative and that allow us to make comparative statements among people as well as to determine whether a person has a headache. These properties are a reflection of the fact that the measurements have not only a constant unit, but also a (absolute) zero point that reflects the absence of what is being measured. Invoking Stevens s (1946) levels of measurement taxonomy or Coombs s (1974) taxonomy these numbers would reflect a ratio scale. In contrast, if we use a GSR device for measuring anxiety we would need to establish a baseline or a zero point by canceling out an individual s normal skin resistance static level before we measure the person s GSR. As a result, and unlike that of the ratio scale, this zero point is not an absolute zero, but rather a relative one. However, all of our measurements would still have a constant unit and would be considered to be on an interval scale. Another approach to measuring anxiety is to ask an individual to rate his or her anxiety in terms of severity. This ratings approach would produce numbers that are on an ordinal scale. These approaches allow us to make comparative statements, such as This person s anxiety level is greater than (or less than) that of another, or in the case of the interval scale, This person s anxiety level is half as severe as that person s anxiety level. Alternatively, if our question simply requires the respondent to reply yes, he or she is experiencing a symptom, or no, he or she is not, then the yes/no responses would reflect a nominal scale. These various scenarios show that how we interpret and use our data needs to take into account the different types of information that the observations carry. In the following discussion we present three approaches for establishing a correspondence between our observations and our latent variable. We begin by briefly introducing IRT, followed by classical test theory (CTT). Both of these approaches assume that the latent variable is continuous. The last approach discussed, latent class analysis (LCA), is appropriate for categorical latent variables. Appendix E, Mixture Models, addresses the situation when a latent variable is conceptualized as having categorical and continuous facets. Item Response Theory Theory is used here in the sense that it is a paradigm that attempts to explain all the facts with which it can be confronted (Kuhn, 1970, p. 18). IRT is, in effect, a system of models that defines one way of establishing the correspondence between latent variables and their manifestations. It is not a theory in the traditional sense because it does not explain why a person provides a particular response to an item or how the person decides what to answer (cf. Falmagne, 1989). Instead, IRT is like the theory of statistical estimation. IRT uses latent characterizations of individuals and items as predictors of observed responses. Although

5 Introduction to Measurement 5 some researchers (e.g., Embretson, 1984; Fischer & Formann, 1982) have attempted to use item characteristics to explain why an item is located at a particular point, for the most part, IRT like other scaling methods (e.g., Guttman Scalogram, Coombs Unfolding) treats the individual as a black box. (See Appendix E, Linear Logistic Test Model [LLTM], for a brief presentation of one of these explanatory approaches, as well as De Boeck & Wilson [2004] for alternative approaches.) The cognitive processes used by an individual to respond to an item are not modeled in the commonly used IRT models. In short, this approach is analogous to measuring the speed of an automobile without understanding how an automobile moves. 1 In IRT persons and items are located on the same continuum. Most IRT models assume that the latent variable is represented by a unidimensional continuum. In addition, for an item to have any utility it must be able to differentiate among persons located at different points along a continuum. An item s capacity to differentiate among persons reduces our uncertainty about their locations. This capacity to differentiate among people with different locations may be held constant or allowed to vary across an instrument s items. Therefore, individuals are characterized in terms of their locations on the latent variable and, at a minimum, items are characterized with respect to their locations and capacity to discriminate among persons. The gist of IRT is the (logistic or multinomial) regression of observed item responses on the persons locations on the latent variable and the item s latent characterizations. Classical Test Theory Like IRT, classical test theory (CTT) or true score theory also assumes that the latent variable is continuous. CTT is the approach that most readers have been exposed to throughout their education. In contrast to IRT in which the item is the unit of focus, in CTT the respondent s observed score on a whole instrument is the unit of focus. The individual s observed score, X, is (typically) the unweighted sum of the person s responses to an instrument s items. In ability or achievement assessment this sum reflects the number of correct responses. CTT is based on the true score model. This model relates the individual s observed score to his or her location on the latent variable. To understand this model, assume that an individual is administered an instrument an infinite independent number of times. On each of these administrations we calculate the individual s observed score. The mean of the infinite number of observed scores is the expectation of the observed scores (i.e., µ i = E(X i )). On any given administration of the instrument the person s observed score will not exactly agree with the mean, µ, of the observed scores. This difference between the observed score and the mean is considered to be error. Symbolically, we may write the relationship between person i s observed score, the expectation, and error as X i = µ i + Ε i (1.1) where Ε i is the error score or the error of measurement (i.e., Ε i = X i µ i ); Ε is the capital Greek letter epsilon. Equation 1.1 is known as the true score model. In words, this model states

6 6 THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY that person i s observed performance on an instrument is a function of his or her expected performance on the instrument plus error. Given that the error scores are considered to be random and that µ i = E(X i ), then it follows that the mean error for an individual across the infinite number of independent administrations of the instrument is zero. By convention µ i is typically represented by the Latin or Roman letter T. However, to be consistent with our use of Greek letters to symbolize parameters, we use the capital Greek letter tau, Τ. The symbol Τ represents the person s true score (i.e., Τ i = E(X i )). The term true score should not be interpreted as indicating truth in any way. As such, true score is a misnomer. To avoid this possible misinterpretation, we refer to Τ i (or µ i ) as individual i s trait score. 2 A person s trait score represents his or her location on the latent variable of interest and is fixed for an individual and instrument. The common representation of the model in Equation 1.1 is X i = Τ i + Ε i (1.2) Although Equation 1.1 may be considered more informative than Equation 1.2, we follow the convention of using Τ for the trait score. There is a functional relationship between the IRT person latent trait (θ) and the CTT person trait characterization. This relationship is based on the assumption of parallel forms for an instrument. That is, each item has the same response function on all the forms. Following Lord and Novick (1968), assume that we administer an infinite number of independent parallel forms of an instrument to an individual. Then the expected proportion of 1s or expected trait score, EΤ, across these parallel forms is equal to the average probability of a response of 1 on the instrument, given the person s latent trait and an IRT model. As a consequence, the IRT θ is the same as the expected proportion EΤ except for the difference in their scales of measurement. That is, θ has a range of to, whereas for EΤ the range is 0 to 1. The expected trait score EΤ is related to the IRT latent trait by a monotonic increasing transformation. This transformation is discussed in Chapters 4 and 10. In addition to the true score model, CTT is based on a set of assumptions. These assumptions are that, in the population, (1) the errors are uncorrelated with the trait scores for an instrument, (2) the errors on one instrument are uncorrelated with the trait scores on a different instrument, and (3) the errors on one instrument are uncorrelated with the error scores on a different instrument. These assumptions are considered to be weak assumptions because they are likely to be met by the data. In contrast, IRT is based on strong assumptions. 3 These IRT assumptions are discussed in the following chapter. These CTT assumptions and the model given in Equation 1.1 (or Equation 1.2) form the basis of the psychometric concept of reliability and the validity coefficient. For example, the correlation of the observed scores on an instrument and the corresponding trait scores is the index of reliability for the instrument. Moreover, using the variances of the trait scores ( σ 2 T ) and observed scores ( σ 2 X ) we can obtain the population reliability of an instrument s scores: ρ XX = σ σ 2 T 2 X (1.3)

7 Introduction to Measurement 7 Because trait score variance is unknown, we can only estimate ρ XX. Some of the traditional approaches for estimating reliability are KR-20, KR-21, and coefficient alpha. An assessment of the variance of the errors of measurement in any set of observed scores may be obtained by substituting σ 2 T = σ 2 X σ 2 E into Equation 1.3 to get σ 2 2 E = σx( 1 ρxx ) (1.4) The square root of σ 2 E (i.e., σ E )is referred to as the standard error of measurement. The standard error of measurement is the standard deviation of the errors of measurement associated with the observed scores for a particular group of respondents. From the foregoing it should be clear that because an individual s trait score is latent and unknown, then the error associated with an observed score is also unknown. Therefore, Equations 1.1 and 1.2 have two unknown quantities, an individual s trait score and error score. Lord (1980) points out that the model given in Equations 1.1 or 1.2 cannot be disproved by any set of data. As a result, one difference between IRT and CTT is that with IRT we can engage in model data fit analysis, whereas in CTT we do not examine model data fit and simply assume the model to be true. As may be obvious, the observed score X is influenced by the instrument s characteristics. For example, assume a proficiency testing situation. An easy test administered an infinite number of independent times to an individual will yield a different value of Τ i than a difficult test administered an infinite number of independent times to the same individual. This is analogous to the example of measuring the shoe and cereal boxes by using the shortest dimension of each box. In short, as is the case with the box example, in CTT person measurement is dependent on the instrument s characteristics. Moreover, because the variance of the sample s observed scores appears in both Equations 1.3 and 1.4, one may deduce that the heterogeneity (or lack thereof) of the observed scores affects both reliability and the standard error of measurement. Moreover, Equations 1.3 and 1.4 cannot be considered to solely be properties of the instrument, but rather also reflect the sample s characteristics. In short, the instrument s characteristics affect the person scores and sample characteristics affect the quantitative indices of the instrument (e.g., item difficulty and discrimination, reliability, etc.). Thus, Thurstone s (1928) idea of invariance does not exist in CTT. In contrast, with IRT it is possible to have invariance of both person and item characterizations. See Appendix E, Dependency in Traditional Analysis Statistics and Observed Scores, for a demonstration of this lack of invariance with CTT. In addition, Gulliksen (1987) contains detailed information on CTT and Engelhard (1994, 2008) presents a historical view of invariance. Latent Class Analysis Unlike IRT s premise of a continuous latent variable, in latent class analysis (LCA) the latent variable is assumed to be categorical. That is, the latent variable consists of a set of mutually exclusive and exhaustive latent classes. 4 To be more specific, there exists a set of latent classes, such that the manifest relationship between any two or more items on a test can be

8 8 THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY accounted for by the existence of these basic classes and by these alone (Stouffer, 1950, p. 6). In LCA the comparison of individuals involves comparing their latent class memberships rather than comparing their locations on a continuous latent variable. For an understanding of the nature of a categorical latent variable, we turn to two empirical studies. The first is a study of the nosologic structure of psychotic illness by Kendler, Karkowski, and Walsh (1998). In their study, these authors conceptualized this latent variable as categorical. Their LCA showed that their participants belonged to one of six classes: (1) classic schizophrenia, (2) major depression, (3) schizophreniform disorder, (4) bipolarschizomania, (5) schizodepression, and (6) hebephrenia. The second example involves cheating on academic examinations (Dayton & Scheers, 1997); the latent variable is cheating. The LCA of the investigators data revealed a structure with two latent classes. One class consisted of persons who were persistent cheaters, whereas the second class consisted of individuals who would either exhibit opportunistic cheating or might not cheat at all. In both of these examples, we can see that respondents differ from one another on the latent variable in terms of their class membership rather than in terms of their locations on a continuum. In Appendix E, Mixture Models, we discuss an approach in which we combine LCA and IRT. That is, we can conceptualize academic performance as involving both latent classes and continua. For example, we can have one class of persistent cheaters and another class of noncheaters. Within each class there is a proficiency variable continuum. Therefore, the cheater class has its own continuum on which we can compare individual performances. Similarly, the noncheater class has a separate continuum that we use to compare the noncheaters performances with one another. These latter comparisons are not contaminated by the cheaters performances and the noncheaters are not disadvantaged by the presence of the cheaters. In general, LCA determines the number of latent classes that best explains the data (i.e., determining the latent class structure). This process involves comparing models that vary in their respective number of classes (e.g., a one-class model, a two-class model, and so on). Determining the latent class structure involves not only statistical tests of fit, but also the interpretability of the solution. With each class structure one has estimates of the items characteristics. Based on these item characteristics and the individuals responses, the respondents are assigned to one of the latent classes. Subsequent to this assignment, we obtain estimates of the relative size of each class. These relative sizes are known as the latent class proportions (πνs, where ν is the latent class index). The sum of the latent class proportions across the latent classes is constrained to 1.0. For example, if the latent variable, say algebra proficiency, has a two-class structure, then π 1 might equal 0.65 and π 2 = = Moreover, our latent classes interpretation may reveal that the larger class (i.e., π 1 = 0.65) consists of persons who have mastered algebraic problems, whereas the other class consists of individuals who have not mastered the problems. In short, the data s latent structure consists of masters and nonmasters. One may conceive of a situation in which if one had a large number of latent classes and if they were ordered, then there would be little difference between conceptualizing the latent variable as continuous or as categorical. In point of fact, a latent class model with a sufficient number of latent classes is equivalent to an IRT model. For example, for a data set with four items, then a latent class model with at least three latent classes would provide equivalent

9 Introduction to Measurement 9 item characterizations as an IRT model that uses only item location parameters. Appendix E, Mixture Models, contains additional information about LCA. Summary Typically, measurement is viewed as analogous to using a ruler to measure the length of an object. In effect, this is analogous to Stevens s (1946) definition of measurement in that the ruler provides the rules and the numeric values associated with the ruler s tick marks provide the numeric labels. However, Stevens s definition invites misinterpretation. Although one can infer from his definition that he is describing an act or a process, this aspect of the definition is not made salient. Moreover, by focusing only on the assignment of numbers, one is left with the impression that measurement results in only a set of numeric labels. We consider measurement to be a process by which one attempts to understand the nature of a variable by applying mathematical techniques. The result may or may not be numeric labels and may or may not involve a continuous variable. For example, LCA is a measurement paradigm that allows one to understand the nature of a latent variable, such as ethnocentrism or test anxiety, without resulting in numeric labels. The use of LCAs involves the application of mathematical techniques that results in individuals being classified into latent classes and an assessment of how well the class structure describes the manifest data. The term manifest data refers to the information obtained by direct observation, whereas the term latent refers to the information obtained on the basis of additional assumptions and/or by making inferences from the original (manifest) data (Lazarsfeld, 1950). Presumably, one or more latent variables can account for the patterns or relationships that are evident in the manifest data. Therefore, a manifest variable is an observed manifestation of one or more latent (i.e., unobservable) variables. We outlined different paradigms that allow the tools of mathematics to be applied to explaining manifest observations from a latent variable perspective. A latent variable may be conceptualized as continuous, categorical, or some combination of the two. When the variable is conceptualized as continuous, then the use of CTT or IRT may be the appropriate mathematical technique. However, if the variable is conceptualized as categorical, then LCA may be the most appropriate psychometric method to use. It is possible to conceptualize the latent space as a set of latent classes, within each of which there is a continuum, or as a combination of latent classes and a latent continuum. In this situation a mixture of IRT and LCA may be considered an appropriate representation of the latent space. As part of measurement it is necessary to operationalize the variable(s) of interest (i.e., provide operational definitions). The measurement process also involves assessing how much information the measures yield about the participants (e.g., reliability) as well as how well the measures reflect the latent variable(s) (i.e., validity). When IRT is appropriate and when there is model data fit, then IRT offers advantages over CTT. For instance, with IRT our person location estimates are invariant with respect to the instrument, the precision of these estimates is known at the individual level and not just at the group level (as is the case with Equation 1.4), and the item parameter estimates

10 10 THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY transcend the particular sample used in their estimation. Moreover, unlike CTT, with IRT we are able to make predictive statements about respondents performance as well as examine the tenability of the model vis-à-vis the data. In the next chapter the simplest of the IRT models is presented. This model, the Rasch or one-parameter logistic model, contains a single parameter that characterizes the item s location on the latent variable continuum. We show how this single-item parameter can be used to estimate a respondent s location on a latent variable. Notes 1. Although understanding how the automobile moves is not necessary in order to measure the speed with which it moves, nonetheless fully understanding how the automobile moves can lead to an improved measurement process. 2. A trait is exemplified primarily in the things that a person can do (Thurstone, 1947) and is any distinguishable, relatively enduring way in which one individual varies from another (Guilford, 1959, p. 6). However, we do not consider traits to be rigidly fixed or predetermined (see Anastasi, 1983). 3. There is an implicit unidimensionality assumption in CTT. That is, for observed scores to have any meaning they need to represent the sum of responses to items that measure the same thing. For instance, assume that an examination consists of five spelling questions and five single-digit addition problems. Presumably our examination data would consist of two dimensions representing spelling and addition proficiencies. If a person had an observed score of 5, it would not be possible to determine whether he or she is perfect in spelling, perfect in addition, or in some combination of spelling and addition proficiencies. In this case, the observed score has no intrinsic meaning. In contrast, if the examination consists of only spelling questions, then the score would indicate how well a person could spell the questions on the test and would have intrinsic meaning. 4. Both IRT and LCA can be considered to be special instances of the general theoretical framework for modeling categorical variables, known as latent structure analysis (LSA; Lazarsfeld, 1950). Moreover, both linear and nonlinear factor analysis may be regarded as special cases of LSA (McDonald, 1967). Copyright 2009 The Guilford Press. All rights reserved under International Copyright Convention. No part of this text may be reproduced, transmitted, downloaded, or stored in or introduced into any information storage or retrieval system, in any form or by any means, whether electronic or mechanical, now known or hereinafter invented, without the written permission of The Guilford Press. Guilford Publications, 72 Spring Street, New York, NY 10012,