Transcription

1 Paul Paul Barrett Barrett Affiliations: Mariner7 Ltd., Ltd., Auckland NZ NZ Dept. Dept. of of Psychology, Univ. Univ. of of Auckland Dept. Dept. of of Clinical Clinical Psychology, Univ. Univ. Of Of Liverpool International Psychometrics Consultant: The The State State Hospital, UK UK

2 Brian Haig (2002) Towards an abductive theory of scientific method. In N. Stephenson et al (eds), Theoretical Issues in Contemporary Psychology. Boston: Kluwer. Michael Maraun (1998) Measurement as a Normative Practice. Theory and Psychology, 8, 4,

3 Joel Michell (1999) Measurement in Psychology: a critical history of a methodological concept. London: Cambridge University Press Peter Schönemann (1994) Measurement: The Reasonable Ineffectiveness of Mathematics in the social sciences. In I. Borg, and P. Mohler (eds), Trends and Perspectives in Empirical Social Research. Berlin: Walter De Gruyter

4 Phenomena: relatively stable, recurrent general features of the world that we seek to explain Phenomena comprise a varied ontological bag that includes objects, states, processes, events, and other features that are hard to classify Phenomena, not data, are the proper objects of scientific explanation; it is Phenomena that give scientific explanations their point (Brian Haig, 2002). Data: idiosyncratic, ephemeral, and pliable, and serve as observable evidence for phenomena It is from data that we extract or detect phenomena

5 The detection of phenomena can be by any means from simple qualitative observation, probabilistic modelling, experimentation, multivariate statistics, or simple counts. All that is required in this process are standards of reproducibility (replicability).

6 Having identified a phenomenon, the scientist tries to explain the cause/s of it. This is the essence of Haig s theory of abductive science that of exploring causes of established outcomes/phenomena

7 Probabilistic models that do not contain measured attributes are simple descriptions of attributes (occurences, sequences, outcomes) that define phenomena. In and of themselves, they cannot explain or give meaning to the causes of phenomena as they merely index the conditions under which outcomes/phenomena will be observable, or not.

8 A simple probability model might index the likelihood of an experiment assistant individual quitting Stanley Milgram s shock study.. We can make statements as to the probability of a person quitting at each shock-level, conditional upon the level of shock. We might introduce the conditional feature of the experimenter forcing compliance at certain levels. Either way, it is a simple probability model of experiment assistance compliance. We have no indications in such a model as to why people quit.

9 If we observe students arriving and departing from a set {a, b, c, d, and e} of cafés on-campus during an academic day, we could obtain probability estimates of the likelihood of n students visiting {a, b, c, d, or e} conditional upon time of day. Let us assume one café out of all others has a higher probability of being visited than the others at any time of day. The model says nothing about why.

10 The question is: Beyond qualitative statements, what is required to enable the investigation (understanding) of the causes of identified phenomena?

11 In order to understand cause beyond simple qualitative statements, meaning-laden theory and units of measurement for every variable entering the causal nexus of the theory are required

12 the likelihood of an experiment assistant individual quitting Stanley Milgram s shock study.. If we hypothesise that it is internal conflict and discomfort at having to hurt people.. then we might measure autonomic arousal (blood pressure, heart rate, respiration rate, fidget-level, muscular tension, skin conductance etc.) showing perhaps that an explanation of the causes of their leaving the experiment is due to exceeding a certain threshold of effects (relative to an individual baseline) in conjunction with certain levels of personality characteristic scores such as autonomy.

13 Unit-less or plastic units over the magnitude range of a measurement scale constrains causal understanding. As the concatenation unit approaches a standard value, so does the possibility of explaining cause more precisely increase.

14 Using Additive Conjoint Measurement By Fiat By Implication as in Classical Test Theory

15 But, the world of psychometric psychology is littered with latent traits and latent variables, using a variety of the unit-creation strategies above. Yet, as in personality psychology and differential psychology, our knowledge concerning the causal basis for the phenomena defined by the various psychometric models is probably little more than it was 50 years ago. Why?

16 Empirically identifying standard or even nonstandard measurement units alone is not sufficient for the causal explanation of phenomena. Remember Rozeboom s liquids. There is another process that is required to complete the process that of establishing a normative meaning for each variable associated with a phenomenon, such that measurement no longer takes place in a theoretical vacuum.

17 It is now worth standing back for a moment and looking at the conclusions of four key individuals

18 Roderick McDonald (1999) Test Theory: A Unified Treatment. New York: Lawrence Erlbaum In typical applications then, a measurement rule does not need to yield scale values corresponding to equal differences in the attribute, to satisfy the requirements fo a scale. We can have number-right, or number-keyed scores, or total or mean scores from quantitative items, or convenient transformations of these to origins and units based upon some population of interest, or possibly nonlinear transformations of them. Numbers are assigned to the observations according to a rule of correspondence, giving us scale values. (p.61)

19 Joel Michell (1999) Because measurement involves a commitment to the existence of quantitative attributes, quantification entails an empirical issue: is the attribute involved really quantitative or not? If it is, then quantification can sensibly proceed. If it is not, then attempts at quantification are misguided. A science that aspires to be quantitative will ignore this fact at its peril. It is pointless to invest energies and resources in the enterprise of quantification if the attribute involved is not really quantitative. The logically prior task in this enterprise is that of addressing this empirical issue. I call it the scientific task of quantification. (p.75)

20 Peter Schönemann (1994) Campbell never waffled on the basic question: Why measure anyway? According to him Measurement [is] only a means to an end: we want to express the properties of systems by numerals only because we are thereby enabled to state laws about them. (Campbell, 1920/1957). The modern measurement theorists, on the other hand, became more and more disoriented about the ultimate purpose of their axiomatic systems as time went by, until in the end, numerical assignments became entirely dispensable. At this point it is no longer clear what purpose measurement is supposed to serve. (p. 152/3)

21 Peter Schönemann (1994) What is left is a redefinition of fundamental measurement and eventually of measurement itself. According to Luce(1993) in areas in which [axiomatic measurement theory] has had considerably more impact than Cliff(1992) acknowledges it assumes the form of theory, not scale construction (p.127) What should have been self-evident from the start is that a research strategy which develops models independently as a body of abstract formal theory with empirical interpretations being left to a later stage was doomed from the outset (p. 158)

22 Michael Maraun (1998) Measurement practice in psychology misdiagnoses the nature of measurement, since it is uniformly formulated under the assumption that measurement claims are justified in large part through empirical case-building [aka construct validity] (p. 436) The problem is that in construct validation theory, knowing about something is confused with an understanding of the meaning of the concept that denotes that something..

23 Scientifically speaking, to make clear what something is means to set forth the laws in which it occurs. (Cronbach and Meehl, 1955) Michael Maraun (1998) This is mistaken. One may know more or less about it, build a correct or incorrect case about it, articulate to a greater or lesser extent the laws into which it enters, discover much, or very little about it. However, these activities all presuppose rules for the application of the concept that denotes it (e.g. intelligence, dominance). Furthermore, one must be prepared to cite these standards as justification for the claim that these empirical facts are about it. (p. 448)

24 Schönemann is correct, with a simple caveat axiomatic measurement theory is irrelevant to phenomena detection. In fact, it is an impediment to it as it places severe constraints upon the detection process which restricts the available search space for a phenomena. McDonald s position is consistent with this view. However, once a phenomenon has been detected, then I agree with Michell to move beyond description of phenomena using ad-hoc quantification or qualitative statements, to the measurement of that phenomenon, requires investigation of the quantitative structure of any variable associated with that phenomenon

25 Maraun s position is the final interlocking piece of this jigsaw whatever measurement is to be created, if at all possible, will need to be created within a normative frame of meaning. That is, it is impossible to create measures of intelligence or depression unless these constructs/phenomena have a normative meaning such that all investigators can work within this common semantic framework. Without this normative agreement, as is the case today, chaos reigns as measure after measure is produced but with no common units or unambiguous normative meaning

26 In my opinion, William Fisher (1997). Physical disability construct convergence across instruments: Towards a universal metric. Journal of Outcome Measurement, (2), has it exactly right when he argues for the equivalent of a System International series of standard unit measures within psychology. This is part of the normative process that Maraun postulates i.e. the rules which are constitutive of the measurement of a construct. The key here is the concept of metrology aligned to the a priori normative definition of a construct or concept.

27 But, note Maraun (1998) again The relative lack of success of measurement in the social sciences as compared to the physical sciences is attributable to their sharply different conceptual foundations. In particular, the physical sciences rest on a bedrock of technical concepts, whilst psychology rests on a web of common-or or- garden psychological concepts. These concepts have notoriously complicated grammars [of meaning]. (p. 436)

28 The Lexile-unit scale of reading proficiency, used throughout the state of Florida, USA. The unit is probabilistically equal-interval, created using the Rasch measurement model. from Ben Wright (personal communication) About Lexiles...they are theory based. To read one must know the words and parse the sentence. Word frequency proxies word familiarity. Sentence length proxies parsing. The Lexile theory item difficulties match their empirical calibrations so closely that one is encouraged to read Lexile measures as about something fundamental - like heat.

29 From Jack Stenner, the creator (personal communication).. In the Lexile Framework for Reading [proficiency] ( item calibrations come from theory and these calibrations embody our intentions regarding the reading variable independent of the person response data. Person fit is at once a test of the quantitative hypothesis (Michell, 1999) and the substantive construct theory. Good fit over 10,000's persons, different item formats and different demographic and age groupings means that the Lexile Theory tells a useful story about what reading is.

30

31 the concept is technical Reading Proficiency the concept is defined normatively by theory the measurement model follows normative rules so that failure to obtain the expected outcome demonstrates a failing in the normative-basis of the concept simultaneous with a failure to demonstrate the concept is quantitative.

32 When someone who cannot read is measured using the scale, they will obtain to all intents and purposes, a 0 score. That is, they have no measurable reading proficiency. This is exactly what is required from a quantitative measure. It is not relative to some group, or some subsample it is absolute.

33 Is this a technical construct? No So, we reject it outright as a construct which can be given any kind of normative meaning. As simple, clean, and straightforward as that. Exactly what Jensen has done in his book Jensen, A.R. (1998) The g Factor: The science of mental ability. Westport, Connecticut: Praeger

34 For technical concepts read cognitive abilities, then even cognitive proficiencies? So, we have say a concept of eduction of relations.. Proposed conceptually by Spearman and later measured by John Raven a series of items in which patterns of occurrence are required to be elucidated such that a decision can be reached about the expected next image in a sequence of images. Well defined theory, clear meaning statement --- IF we solely stay within our technical definition.

35 David Andrich has scaled the items using the Rasch measurement model. We have our quantitative scale of eduction of relations on which we can place individuals. No problems so far. We can do this for Vocabulary attainment, arithmetic proficiency, symbol usage (Digit Symbol), Perceptual Speed (Letter Cancellation), Spatial Reasoning (paper folding) etc.

36 Spearman s g as a technical concept well defined, highly technical definition of a phenomena which is identified using covariance analysis. BUT psychologists have tried to map a common-or garden concept of intelligence onto this g with disastrous results. Suddenly specific ability tests, created using a technical concept specification, have been retro-fitted onto an undefined concept of intelligence. Chaos ensues.

37 We have several nonsenses following from this: The IQ variable is created which is constructed adhoc from one or more of the constituent technical concepts. IQ is taken as a measure of intelligence by some, but not by others. This is because the meaning of IQ is conflated with specific technical concepts and then mapped onto an arbitrary concept of intelligence. What s worse, the biometrical geneticists then try and use IQ as a proxy for Intelligence in order to find genes for IQ test scores.

38 See: Capron, C., Vetta, A.R., Duyme, M., and Vetta, A. (1999) Misconceptions of Biometrical IQists. Cahiers de Psychologie Cognitive/Current Psychology of Cognition, 18, 2,

39 In a sense, a normative definition is being aimed for along the lines of.. Intelligence is defined by the g latent variable which is inferred from covariance patterns amongst many different kinds of specific cognitive ability tests. We will call this g intelligence. What is now required is a demonstration that the multitude of available cognitive ability tests can be scaled onto g using an additive concatenation function, as per lexile scale. In short, Michell s quantitative hypothesis is required to be tested.

40 This is not an easy task, but it is essential. For without an empirically established metric standardised scale of g, that is taken as the normative measurement scale on which all cognitive proficiency tests may be located, then we remain as we are, with hundreds of ability scales each of which provides IQ scores, but with no clarity as to which IQ score is the more valid IQ score than any other test.

41 Why does any of this cause a problem.. well, its because many psychologists have bolted before the stable door was closed. The normative definition of intelligence was never completed. It is suggested here that this will remain an impossible ideal unless the definition is restricted to a solely technical form. Hence, the almost irrelevant exercise of 50 or more psychologists each generation to keep re-defining what is intelligence as though this will in any way solve the basic problems of technical-conceptual clarity and the quantitative hypothesis.