Craft Personality Questionnaire Evidence of Reliability and Validity 888-298-6227 TalentLens.com Copyright 2008 Pearson Education, Inc., or its affiliates. All rights reserved.
Copyright 2008 by Pearson Education, Inc., or its affiliate(s). All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the copyright owner. Pearson and TalentLens logos, and CPQ are trademarks, in the U.S. and/or other countries, of Pearson Education, Inc. or its affiliate(s). Portions of this work were previously published. Printed in the United States of America.
Introduction The Craft Personality Questionnaire (CPQ ) is a factor-analytically derived assessment of personality traits known to be predictive of work performance, particularly in careers such as sales, management, customer service, call center, and related positions. The development of the CPQ traits took place after exhaustive research involving a team of psychologists, social scientists, and statistical experts. Over 300,000 participants were assessed at various stages of the CPQ s development cycle, providing sufficient data to ensure that the tool is fair, valid, reliable, and job-related for use in employment testing. Reliability Reliability, as it is defined in psychometrics, pertains primarily to the relative consistency of the questionnaire s results over time. In addition, it is an indication of the predictable range of fluctuation that is likely to occur in an individual s score due to chance or irrelevant factors. The CPQ s construction process placed a heavy emphasis upon the internal consistency and reliability of the questionnaire s 75 items. Initially, hundreds of questions were reduced to their common denominators and then grouped using both inductive and deductive methodologies. Informal traits were developed with approximately 10 items in each trait. Factor and reliability analyses helped to eliminate weakly worded questions that were contributing to error variance. These procedures firmly established the traits as stable descriptors of trait behavior. A sample of 309,603 individual item responses was used for reliability analyses, with the results shown below. Reliability Results Trait Cronbach s Guttman Reliability Statistics (Lambda) Alpha L1 L2 L3 L4 L5 L6 Goal-orientation.76.68.77.76.61.74.77 Need for Control.74.66.75.74.72.73.74 Social Confidence.79.70.80.79.75.78.78 Social Drive.82.73.82.82.81.79.81 Detail-orientation.79.71.79.79.80.77.79 Good Impression.72.65.73.72.74.71.71 Need to Nurture.75.67.76.75.76.74.75 Skepticism.75.68.76.75.60.74.75 As statistical experts have contended, to base assumptions of reliability on a single administration of a test is to ignore the potential variance that can come from repeated administrations. Test-retest reliability is important to safeguard what some refer to as the repeatability of a test. Specifically, test-retest procedures measure the stability and consistency of the traits over time. Measuring the variance found from time sampling is critically important in determining how susceptible the traits are to random daily fluctuations. Specifically, the user must be confident that the traits will provide results that do not fluctuate substantially due to mood, fatigue, or other environmental factors that may have been present during one administration but not another. Test-retest reliability measures are the most obvious choice for providing the user with this level of confidence. 1
One study sought to develop an understanding of the reliability associated with a shorter duration between two administrations. From a practical view, managers are often confronted with a candidate that, for one reason or another, asserts they need a second opportunity to take a pre-employment questionnaire such as the CPQ. Perhaps they assert that the environment was less than pleasant, that they were rushed, or that they did not fully understand the instructions. For these and other reasons, managers have often looked to psychometric research for some advice as to whether a second administration, done only days later, could produce substantially different results. To answer that question, a study of short duration retest reliability was performed on a sample of office equipment sales representatives. These participants (n = 72) took the CPQ and without any indication of their results, were permitted to retake the CPQ approximately 1 day later. The aforementioned short duration analysis was repeated for participants (n = 120) with up to 5 days of elapsed time between administrations. This analysis was again performed for participants (n = 195) with up to 15 days of elapsed time between administrations. Test-Retest Reliability Results Trait 1 day trial 5 day trial 15 day trial Goal-orientation.79.74.75 Need for Control.80.79.75 Social Confidence.60.65.66 Social Drive.82.77.78 Detail-orientation.86.82.80 Good Impression.74.74.69 Need to Nurture.75.75.70 Skepticism.76.72.71 Throughout the worldwide utilization of the CPQ, various organizations have provided multiple questionnaire results for participant samples. These samples have included participants who, for a variety of reasons, took the CPQ on more than one occasion throughout their job tenure. Most often, the participant took the CPQ during training and development exercises (as an incumbent), several months after taking the CPQ during a selection process (as a candidate). These samples are taken from uncontrolled settings, and the results derived indicate reliability evidence obtained from practical applications. The findings are from the financial services industry and are highlighted in the following table. 2
Test-Retest Reliability Results Avg. 37 weeks (n = 61) Avg. 2 years (n = 54) Position Account Executives Sales/Agency Managers Goal-orientation.70.74 Need for Control.59.82 Social Confidence.68.71 Social Drive.72.76 Detail-orientation.69.76 Good Impression.79.77 Need to Nurture.71.59 Skepticism.59.66 Validity Validation is a scientific process of systematically collecting, analyzing, and summarizing data to provide evidence that score interpretations can support an intended measurement. A common misinterpretation is that an assessment itself is valid (or not valid), when in fact the actual assessment is not validated. Rather, validity represents the inferences that can be made from the assessment in the form of traits and selection models (multiple traits scored together based on a particular job description). There are three methods for establishing validity: Construct validation Criterion validation Content validation The Uniform Guidelines on Employee Selection Procedures (1978) states that criterion validation is the preferred method where technically feasible. The other two approaches contribute to the collection of validity evidence. Construct Validity For this analysis, the eight CPQ traits were measured for correlation with the traits of other instruments used in employment selection and development settings. These instruments may measure similar but not identical constructs, each offering a unique assessment of personality. It is important to note that correlation with other traits that are quite low may suggest the absence of commonality (divergent validity) in measuring a particular construct. In contrast, correlation with other traits that are quite high may suggest a mere duplication (convergent validity) of an existing construct. A moderate degree of correlation is expected for constructs of a similar, but not identical nature. The following table shows the correlation of traits within the CPQ (n = 308,868). 3
CPQ Inter-Trait Correlations Trait Go Nc Sc Sd Do Gi Nn Sk Goal-orientation 1.00 Need for Control.25** 1.00 Social Confidence.05**.31** 1.00 Social Drive.23**.31**.19** 1.00 Detail-orientation -.20** -.15** -.04** -.10** 1.00 Good Impression -.16** -.03**.27** -.09**.28** 1.00 Need to Nurture -.22** -.31** -.07* -.14** -.00*.06** 1.00 Skepticism.14**.04** -.19** -.02** -.14** -.35** -.09** 1.00 * p <.05 The tables that follow demonstrate the correlation of the CPQ traits with other published assessments. The samples utilized for these analyses included participants from a variety of industries and positions. The CPQ and the Myers-Briggs Type Indicator (MBTI) n = 39 MBTI Type Do Gi Go Nc Nn Sc Sd Sk Extraversion -.49**.39*.30^ Introversion.49** -.39* -.30^ Sensing.36* -.38* -.27^ Intuition -.36*.38*.27^ Thinking.27^.28^ -.53** Feeling -.27^ -.28^.53** Judging.49** -.35* Perceiving -.49**.35* * p <.05 ^ p <.10 The CPQ and the DiSC Profile n = 38 DiSC Do Gi Go Nc Nn Sc Sd Sk (D) Dominance -.30^ -.34*.37*.65** -.55** (i) Influence.39*.34*.33* (S) Steadiness.39*.40* -.53** -.57**.46** -.32* (C) Conscientiousness.33* -.33* * p <.05 4
The CPQ and the Occupational Personality Questionnaire (OPQ32i) n = 35 OPQ32i Traits Do Gi Go Nc Nn Sc Sd Sk Persuasive -.33^.31^.39*.46** Controlling -.34*.69** -.40*.55** Outspoken.32^ -.34*.38* Independent Minded -.37* -.28^ Outgoing.58** Affiliative.43** Socially Confident.64** Modest.32^.40* -.38* -.36* -.39* -.47** Democratic -.52**.42* -.35* Caring -.51**.80** -.40* Data Rationale.62** -.40* Evaluative -.32^ Behavioral.31^.32^ -.29^ Conventional.31^ -.36* -.37* -.49** -.42* Innovative.36*.30^ Variety Seeking -.34* Forward Thinking.29^ -.29^ Detail Conscious.51** -.30^ Conscientious -.38* Rule Following -.54** -.62** -.30^ Relaxed -.33^.31^ Worrying -.39*.31^ -.73** Trusting -.46**.51** -.44** Emotionally Controlled -.38* -.48** Vigorous.33^ Competitive -.42* -.37*.31^.55**.28^ Achieving.42*.29^ Decisive -.52**.38*.33^ Consistency -.39* * p <. 05 ^ p <.10 5
The CPQ and the 16 Personality Factors (16PF) n = 35 16PF Traits Do Gi Go Nc Nn Sc Sd Sk Infrequency.31^ Impression Management.77** -.42* -.28^ Extraversion.45** -.33* Anxiety -.34* -.45**.47**.39* Tough Mindedness -.46** Independence.51**.75**.33^ Self Control.46** -.29^ -.52** Warmth.39* Reasoning.28^ Emotional Stability.36* -.44** Dominance.53**.67**.32^ -.29^ Liveliness.34*.44**.41* -.33* Rule Conscious.29^ Social Boldness.41*.64**.37* Sensitivity.33* Vigilance -.45** -.46**.43** -.30^ Abstractedness -.41* -.50**.30^.54** Privateness.30^ -.35* Apprehension -.33^.44** Openness to Change.48** Self Reliance -.31^ Perfectionism.44**.48** -.50** Tension -.34*.36* -.38* * p <.05 ^ p <.10 6
The CPQ and the Personal Characteristics Inventory (PCI) n = 35 PCI Traits Do Gi Go Nc Nn Sc Sd Sk Agreeableness.51** -.34*.46** -.37* Conscientiousness.42**.60** -.43* -.44** Extraversion.65** -.32^.73**.61** Openness.54**.29^ Stability.44** -.34*.36* -.48** Consideration.58** -.38*.44** -.28^ -.37* Cooperation.30^ -.36*.50** -.32^ Efficiency.44**.37* -.39* Dependability.41*.52** -.38* -.45** Achievement.31^.59**.46** -.57** Leadership.77** -.44*.74**.71** Recognition -.29^ -.32^.61** -.30^.58**.68** Sociability.52**.30^ -.30^ Creative Thinking.46**.31^ Abstract Thinking.53** * p <.05 ^ p <.10 7
Criterion Validity Throughout the development cycle, the CPQ items went through hundreds of iterations aimed at producing a valid and reliable instrument. With numerous validity studies to its credit, the CPQ has been proven to predict performance in specific positions in a variety of industries. In some cases, models were developed using individual CPQ traits that correlated at statistically significant levels to performance. In most cases however, models are developed using position-specific scoring algorithms that weight the CPQ traits according to their statistical power in predicting job performance. Regardless of the methodology employed, the CPQ has provided consistent, predictive results for clients in numerous industries and cultures. The CPQ traits, while a reliable and important measure of an individual s personality traits on their own, are typically evaluated in comparison to an ideal model. The model is used to compare an individual s CPQ traits in relation to the needs of a specific position (for example, sales professional), for which research suggests certain ranges of each personality trait are preferred. The research used in deriving each model differs based on environmental factors, sample availability, and issues related to criteria. The following table highlights a number of local validation studies, and their related criterion validity findings. More information regarding these, or the dozens of other studies conducted, can be obtained by contacting Pearson. 8
Criterion Validation Study Results Position n Criteria Validity Coef. Account Executive 125 134 Career Revenue.26** Monthly Revenue.32** Advertising Sales Manager 32 Gross Earnings.52* Advertising Sales Rep. 132 Gross Earnings.33* Career Production.43** Agency Manager 50 Recruits.37** Unit Count.46** Performance Rating.20 Case Worker 38 40 Competency Rating.32* Restraints -.38* Child Care Director 30 31 Performance Rating.48** Competency Rating.59** Office Equipment Sales 364 456 Survival 1 year.14** Survival 2 years.11* Refinancing.64** Modifications.48** Loan Officer 29 Applications.63** Calls.49** Supervisor Rank.36* Real Estate Sales Associate 41 55 Total Sales Volume.40** Gross Commissions.36** Licenses.19** Sales Manager 221 263 Commissions.15* Premiums.16** Technical Product Sales 50 60 Annual Performance Factor.39** Single Sites Sold.35** Telephone Sales Rep. 51 Sites Sold Per Hour.30* Vacation Packages Sold.31* Disability Income Specialist 69 Annual Production.29* Financial Advisor 130 Cumulative Earnings.56** Resort Sales Rep. 74 Sales Volume Per Guest.41** Housekeeper 154 Supervisor Rating.19* Maintenance Worker 86 Supervisor Rating.27* First Call Resolution.33** Call Center Associate 138 Conversion to a Sale.19* Customer Service Time.37** Work Order Accuracy.17* * p <.05 Content Validity Hiring managers use the CPQ to measure personality traits important to job performance in a variety of employment settings. Within each industry, a given position may require a specific combination of personality traits. Research shows that the general domain of personality, as measured by the CPQ s Basic Eight personality traits, is a valid predictor of employee performance in a variety of employment settings. Content validity is further promoted by a thorough review of job requirements for which the CPQ provides a measure of job compatibility. 9