Download from:

Transcription

1 Paul Barrett 11, and Nina Ebbeling 22 1 Psytech 1 Psytech International International Ltd. 1 Ltd. University 1 University of of Auckland 1 Auckland University 1 University of of Canterbury Canterbury The The Grange Grange Faculty Faculty of of Business Business Department Department of of Psychology Psychology Church Church Road, Road, Pulloxhill Pulloxhill Commerce Commerce C Building Building Private Private Bag Bag Bedfordshire Bedfordshire Symonds SymondsStreet Street Christchurch, Christchurch, NZ NZ MK45 MK45 5HE, 5HE, UK UK Auckland, Auckland, NZ NZ 2 Works 2 Works Infrastructure Infrastructure Amelia Amelia Earhart EarhartAve. Airport Airport Oaks Oaks P.O. P.O. Box Box Auckland, Auckland, NZ NZ Download from:

2 The questionnaire was 1940s and 1950s psychologists response to the problem of assessment, given the technology of that era. We reached the limits of precision of this technology some years ago. Test score and item response psychometrics is increasingly becoming disconnected from psychological reality, meaning instantiation, and the fundamental logic of quantitative measurement.

3 (Sternberg & Williams, 1998; p. 577) "No technology of which we are aware -- computers, telecommunications, televisions, and so on -- has shown the kind of ideational stagnation that has characterized the testing industry. Why? Because in other industries, those who do not innovate do not survive

4 In the testing industry, the opposite appears to be the case. Like Rocky I, Rocky II, Rocky III, and so on, the testing industry provides minor cosmetic successive variants of the same product where only the numbers after the names substantially change. These variants survive because psychologists buy the tests and then loyally defend them.

5 Michell,, J. (2000), p It is concluded that psychometrics is a pathology of science Michell,, J. (2001), p the way in which psychometrics is currently, typically taught actually subverts the scientific method

6 In which areas have the substantive innovations in major psychological tests and assessment methodologies taken place since the late 1950s? Is it mainly in the technology of questionnaire test delivery allied to new measurement models for questionnaire data -orthe innovation of completely new kinds of psychological construct assessment?

7 Was Blinkhorn (1997) correct in his recent review of 50 years of Test Theory? Contemporary test theory, with its emphasis on statistical rather than psychological models, has become inaccessible to the majority of test users, and predominantly reflects educational rather than psychological concerns. Real progress may depend on the emergence of a new and radical reconceptualization.

8 p. 196 The New Psychometrics The original purpose of psychometrics was to provide the precise quantification necessary for science. As we can see, it has failed to do this, although it has provided us with promising results. From the arguments developed in this book, it can be concluded that it is pointless to attempt to develop further psychometric tests of the traditional kind.

9 Think about Psychometrics as Applied Numerics (Barrett, 2003), quantitative science, and non-quantitative science. Innovate.

10 Is it possible to assess an individual s personality as one might with a conventional questionnaire that assesses traits.. without using such a questionnaire?

11 Construct a conventional personality trait questionnaire with known item and scale properties, and high internal-consistency (> 0.75).

12 Using the free 5-factor personality model itembank at the International Personality Item Pool Specifically using 10 facets taken from the AB5C 45-facet personality questionnaire as typical personality test scales, spanning about 106 questionnaire items 106 questionnaire items in total

13 Factor Factor 1: Extraversion Leadership Facet Friendliness Talkativeness Factor 3: Conscientiousness Efficiency Purposefulness Organization Orderliness Factor 4: Emotional Stability Calmness Impulse-Control Happiness

14 Extract the meaning of all the items in a scale, and compose a single rating statement that seems to best encompass the meaning not only of the scale name, but that embodied within the items.

15 single item rewords?... facet = Talkative H4 H1138 H527 H535 H1150 H536 H531 H769 H548 X212 Do most of the talking. Talk too much. Speak loudly. Make myself the centre of attention. Like to attract attention. Never stop talking. Make a lot of noise. Demand to be the centre of interest. Speak softly. Dislike talking about myself.

16 single item rewords?... facet = Talkative The Rating Statement: I have no problem in talking about almost anything. In fact, I find it hard to stop sometimes, especially if I've become the centre of attention! Frankly, I just like talking with people.

17 Use a dynamic, graphical, computerbased technology. Create an assessment where an individual chooses how much a single statement applies to them on a continuous response scale (no numbers, no Likert points, just anchors).

18 Ask 100 university students (Canterbury University) to complete both the questionnaire and graphical profiler assessment (order counterbalanced), and ask them which they prefer to do, and which they felt most accurately represented their personality.

19 This free working demo program and documentation can be downloaded from the Idanet programs page on my website

20 Completion Times and Preference Questionnaire: about 20 minutes Profiler: about 5 minutes 93% of individuals preferred completing the Profiler assessment over the paper and pencil Questionnaire

21 AB5C Scales Profiler Scores Friendliness Leadership Talkativeness Efficiency Purposefulness Profiler -vsquestionnaire scale-score Organization Orderliness Means Calmness Impulse-Control N=99 cases Happiness

22 AB5C Scales Profiler Scores Friendliness Leadership Talkativeness Efficiency Purposefulness Profiler -vsquestionnaire scale-score Organization Orderliness Std. Devns. Calmness Impulse-Control N=99 cases Happiness

23 Friendliness Leadership Talkativeness Efficiency Purposefulness Organization Orderliness Calmness Impulse-Control Happiness Actual Correlation Disattenuated Profiler -vsquestionnaire scale-scores Correlations N=99 cases

24 single item rewords?... facet = Impulse Control H381 H162 H525 H802 E98 H523 H1192 E19 H1176 H764 E68 Keep my emotions under control. Let others finish what they are saying. Demand attention. React intensely. Talk even when I know I shouldn t. Often make a fuss. Shoot my mouth off. Am easily excited. Blurt out whatever comes into my mind. Barge in on conversations. Like to gossip.

25 single item rewords?... facet = Impulse Control The Rating Statement: I rarely express my emotions to others, always careful to think about what I'm going to say before I say it. I really dislike the kind of people who seem unable to properly control their emotions.

26 The profiler response scale encouraged greater range of responding seen clearly when contrasted with their scale scores. That is, given both profiler scores and scale scores are expressed within the same metric as the questionnaire scale scores, we can see that when individuals score low on a questionnaire scale, they score much lower on the profiler scale. Likewise when they obtain high scale scores, they score much higher on the profiler. If we plot the residuals from predicting questionnaire scale scores with profiler scores

27 15 Predicting AB5C Scale "Leadership" by Profiler Score Plotting Prediction Residuals by Profiler Score 10 5 Raw Residuals Personality Profiler Score

28 2. Did you like using the profiler in that you felt you were able to describe yourself adequately on the measured attributes (e.g. talkative, orderly etc.) No, not at all far too limiting -(1%) Just like a questionnaire limiting in what it conveys about you -(12%)

29 2. Did you like using the profiler in that you felt you were able to describe yourself adequately on the measured attributes (e.g. talkative, orderly etc.) Well, not bad actually better than a questionnaire, and probably about as good as this kind of assessment can get (in that it is still a simple assessment of something very detailed i.e. personality) - (87%)

30 Which method is a more accurate assessment of personality? For some discussion of the accuracy issue see Barrett, P.T. and Hutton, R. (2000) Personality and Psychometrics. Selection and Development Review, 16, 2, 5-9 *(downloadable from my preprints/reprints page off my homepage)

31 Car Profiler High Ford Mustang Ford Falcon Performance Ford Laser Mitsubishi Galant Holden Vectra Holden Commodore Mitsubishi Diamante Holden Senator Mitsubishi Chariot Low Maintenance Cost

32 Descriptors Reserved vs Outgoing Unstable vs Calm Stable Me Most Like My Ideal Partner Most Like Humble vs Dominant Wary vs Gushing Sloppy vs Conscientious Shy vs Brash Concrete vs Conceptual Trusting vs Suspicious Implicit vs Explicit Blunt vs Tactful Confident vs Scared - Conventional vs Unconventional Friendly vs Independent - Carefree vs Disciplined Impulsive vs Restrained - Least Like Least Like

33 For the quoted references and more examples of innovative assessments such as Work-Preference Profiling Dynamic Testing Sex-Offender Assessment Download the SIOPSA 2003 keynote presentation on my Presentations Page Psychological Assessment and Data Utility: It s time to innovate.