WORKSHOP Maximum Performance Measures of EI: Consensus and Controversy

Size: px

Start display at page:

Download "WORKSHOP Maximum Performance Measures of EI: Consensus and Controversy"

Marcus Powell
6 years ago
Views:

1 WORKSHOP Maximum Performance Measures of EI: Consensus and Controversy Carolyn MacCann University of Sydney Richard D. Roberts Professional Examination Service

2 Goals of the Workshop Understand a framework for evaluating validity (The test standards) Develop and understand a theoretical framework for evaluating EI Be familiar with some of the major assessments of EI (and similar constructs)

3 Introduction to Maximum Performance Tests of EI

4 Social Intelligence as a Pre-curser to Emotional intelligence (EI) in Psychological Research E.L. Thorndike's (1920) 3 intelligences: ABSTRACT (ideas), MECHANICAL (objects), SOCIAL (people) By social intelligence is meant the ability to understand and manage men and women, boys and girls --to act wisely in human relations David Wechsler (1950s) social intelligence is just general intelligence applied to social situations George Washington Social Intelligence Test (1920s) E.g., Memory for Names and Faces; Observation of Human Behaviour; Recognition of the Mental States Behind Words; Recognition of Mental States from Facial Expression; Social Information; Sense of Humour Factor analysis: Social intelligence is not a separate ability from Abstract intelligence LEGACYImportance of proving EI is separate from existing constructs (personality, Gc, Gf)

A Brief Introduction to Emotional Intelligence (EI) First proposed as measurable ability in 1990 (Mayer, DiPaolo, & Salovey) EARLY DEFINITION the ability to monitor one s own and others feelings and

5 A Brief Introduction to Emotional Intelligence (EI) First proposed as measurable ability in 1990 (Mayer, DiPaolo, & Salovey) EARLY DEFINITION the ability to monitor one s own and others feelings and emotions, to discriminate among them and to use this information to guide one s thinking and actions Remained an obscure research topic until 1995 Daniel Goleman spopular press book HUGE (and growing) popularity Broad ideas rather than a welldefined and assessable construct

6 POPULARIZATION: Effects on the Research...

There are TWO theories of what Emotional Intelligence is Snowballing popularity = many different paradigms: Different theories of EI are quite different 1.

7 There are TWO theories of what Emotional Intelligence is Snowballing popularity = many different paradigms: Different theories of EI are quite different 1. Ability models (four-branch hierarchical model) 2. Mixed models (many; e.g., Trait EI; EQ-i) Different tests of EI are quite different 1. Ability scales [maximum performance] 2. Rating-scales [typical performance] Many fewer Stream 1 tests than other streams Ashkanasy, N. M., & Daus, C. S. (2005). Rumors of the death of emotional intelligence in organizational behavior are vastly exaggerated.journal of Organizational Behavior, 26,

8 Example Stream 1 item Mary was working at her desk. Something happened that caused her to feel surprised. What is most likely to have happened? (a)her work-mate told a silly joke. (b)she was working on a new task she hadn't dealt with before. (c)she found some results that were different from what she thought they would be. ** (d)she realized she would not be able to complete her work. (e)she had to do a task she didn't normally do at work. Example Stream 2 items I have problems dealing with my feelings of anger. [R] When someone I know is in a bad mood, I can help the person calm down and feel better quickly. Example Stream 3 items On the whole, I`m pleased with my life. I feel that I have a number of good qualities MacCann, C., & Roberts, R. D. (2008). New paradigms for assessing emotional intelligence: Theory and data. Emotion, 8, Brackett, M. A., Rivers, S. E., Shiffman, S., Lerner, N., & Salovey, P. (2006). Relating emotional abilities to social functioning: A comparison of selfreport and performance measures of emotional intelligence. Journal of Personality and Social Psychology, 91, Petrides, K. V., & Furnham, A. (2003). Trait emotional intelligence: Behaviouralvalidation in two studies of emotion recognition and reactivity to mood induction. European Journal of Personality, 17,

9 Overview of the MSCEIT NOTE: After seeing the MSCEIT, we will be applying validity standards to it as we go through the validity standards

Four-Branch Hierarchical Model and MSCEIT Mayer, Salovey and Caruso articles in early 1990s Emotional

measure the four-branch model MSCEIT (Mayer-Salovey-Caruso Emotional Intelligence Test) 8 tests, 2 per

Intelligence Reflectively Regulating Emotions Understanding Emotions Assimilating Emotions in Thought

10 Four-Branch Hierarchical Model and MSCEIT Mayer, Salovey and Caruso articles in early 1990s Emotional Intelligence is an ability Should relate to other abilities, show developmental progression Two tests that measure the four-branch model MSCEIT (Mayer-Salovey-Caruso Emotional Intelligence Test) 8 tests, 2 per branch MEIS (Multi-factor Emotional Intelligence Scale) 12 tests, 2-4 per branch John Mayer Emotional Intelligence Reflectively Regulating Emotions Understanding Emotions Assimilating Emotions in Thought Perceiving and Expressing Emotion Strategic Emotional Intelligence Experiential Emotional Intelligence Peter Salovey David Caruso

11 Branch 1: Emotion Perception The ability to accurately perceive emotions and express emotions E.g. to read a person s emotions from their facial expressions; to identify emotions generated by particular natural environments

12 Branch 2: Emotional Facilitation The ability to use emotions to facilitate thought, the ability to generate particular emotions when required E.g. to recognise how different moods impact thinking & decision making; to create the right feelings for facilitating problem-solving

13 Branch 3: Emotion Understanding Label emotion and recognize relations among these labeled emotions. Interpret meanings of emotions and circumstances Understand complex feelings Understand transitions among emotions

14 Branch 4: Emotion Management The ability to manage emotions in the self and others, to recognise the actions that are most effective in repairing negative moods and maintaining positive ones E.g. to select effective emotional strategies for coping with different situations

15 Scoring Performance-Based EI Tests: The problem of the correct answer 1. Consensus scoring[msceit] Large screening sample take test Test scored according to proportion of screening sample choosing a particular option E.g., If 36% of screening sample choose 4 for happiness, selecting 4 will result in a score of 0.36 for the happiness question 2. Expert scoring[msceit] Same as consensus but with group of experts Experts = 21 members of ISRE 3. Target scoring [MEIS only] The target of the item stimuli (artist, model, person who has described emotional incident) decides the correct answer

16 How to evaluate assessments

17 1. Test Content Evidence for Validity Relevance Representativeness MSCEIT SCORECARD? Relevance of facilitation branch? Perception branch is visual only under-represented construct?

18 2. Response Processes as Evidence for Validity If the test is intended to capture a particular process, evidence should show that the test does measure this process Think aloud protocols [Coglabs for test development] Eye-tracking Computer models Susceptibility to manipulations, coaching etc (in line with theory) MSCEIT SCORECARD? Perception of emotion actually perception of NO EMOTION?

19 3. Internal Structure as Evidence for Validity # sub-components found empirically = # subcomponents theoretically expected E.g., 4 components of the MSCEIT model (emotion perception, emotion facilitation, emotion understanding, and emotion management) MSCEIT SCORECARD? Not a lot of support for the facilitation branch Fan, H., Jackson, T., Yang, X., Tang, W., & Zhang, J. (2010). The factor structure of the Mayer Salovey Caruso Emotional Intelligence Test V 2.0 (MSCEIT): A meta-analytic structural equation modeling approach. Personality and Individual Differences, 48(7),

20 4. Relationship to Other Variables as Evidence of Validity Convergent and discriminant validity evidence Relates to similar constructs Unrelated to dissimilar constructs Test-criterion relationships Predicts valued outcomes Validity generalization Replication in different situations Different POPULATIONS(e.g., different countries, states, sectors) Different CONDITIONS (e.g., procotoredversus unproctored, timed versus untimed) Replication for different purposes job performance vs academic achievement performance on different types of jobs MSCEIT SCORECARD? Reasonable Potentially too high with Gc?

21 Variability in observed scores SIDE NOTE: Reliability in CTT Classical Test Theory σ X = σ T + σ E Reliability = Proportion of observed score variance due to true-score variance Variability due to true scores True-score Variance (80%) Error variance (20%) Variability due to measurement error Reliability in Classical Test Theory = σ T /σ X OR = σ T / (σ T + σ E ) E.g., If reliability is.80 80% of the variation in scores is due to the attribute of interest 20% of the variation in scores is random error (chance fluctuations, not meaningful)

SIDE NOTE: Reliability and Validity Under classical test theory, reliabilitysets an upper bound for validity(reliability > validity) Maximum correlation between two measures is set by reliability Max

22 SIDE NOTE: Reliability and Validity Under classical test theory, reliabilitysets an upper bound for validity(reliability > validity) Maximum correlation between two measures is set by reliability Max correlation= EXAMPLE: If test 1 reliability = 0.50 and test 2 reliability = 0.60 Maximum possible correlation = =.54 Can CORRECT FOR RELIABILITY by diving observed correlation by maximum possible reliability Test 1 Test 2 Correlation between tests (reliability of each test = 1.00) Correlation between tests (reliability of tests ~.50 and.60) THIS AFFECTS OUR INTERPRETATION OF THE CORRELATIONS!

23 Evaluating Validity As an ability, EI test scores should be associated with other abilities (i.e., positive correlations on tests of intelligence) BUTnot too high ANDhigher with emotional than non-emotional abilities As an emotion-related ability, EI test scores should relate to differences in emotion-related variables Discussion about what these might be?

24 An Overview/Demonstration of Stream 1 Tests: (1) Emotion Recognition/Perception (2) Higher Branches

25 Emotion Perception/Recognition Tests Diagnostic Analysis of Nonverbal Accuracy (DANVA; Nowicki, 2004; assessment_with_danva.html) Montréal Set of Facial Displays of Emotions (MSFDE; Beaupré & Hess, 2005; Multimodal Emotion Recognition Test (MERT; Bänziger, Grandjean, & Scherer, 2009; Japanese and Caucasian Brief Affect Recognition Test (JACBART; Matsumoto et al., 2000) [example item, more, more] Profile of Nonverbal Sensitivity (PONS) (MiniPONSat SEE ALSO NIMSTIM Stimulus subset ( Reading the Mind in the Eyes Test (

26 Test Characteristics to Note Can be issues with difficulty and therefore variability Reduced variability = reduced reliability, reduced correlations with outcomes Ways to deal with this: Limit the viewing time: Inspection time paradigm; stimulus is shown for a brief period with a forward and backwards mask Use subtle expressions Give partial information(single channel or eyes only) How well do these resemble real life emotional tasks? (and does this matter) Micro-expressions Telephone/ exchanges Is the processassessed by the task (e.g., inspection time) more pertinent than the content(i.e., emotions)? Inspection time is a well known paradigm for mental speed

27 Higher Strategic Branches (Understanding and Management) Situational Test of Emotion Understanding (STEU) Situational Test of Emotion Management (STEM) Interpersonal and Intrapersonal Abilities (Freudenthaler and Neubauer) Emotional Accuracy Research Scale (EARS; example item) Levels of Emotional Awareness Scale (LEAS; Magdeburg Social Intelligence Test EARS: Mayer, J. D., & Geher, G. (1996). Emotional intelligence and the identification of emotion. Intelligence, 22(2), Freudenthaler, H. H., & Neubauer, A. C. (2007). Measuring emotional management abilities: Further evidence of the importance to distinguish between typical and maximum performance.personality and Individual Differences, 42(8),

28 Test Characteristics to Note SJTs can be multidimensional (low internal consistency) Other than test content, many test characteristics can change scores and meaning: Maximum versus typical performance instructions Rate-the-extent versus multiple choice instructions Scoring types (though most converge) Change from text-based to multimedia format can change scores and meaning Multimedia also assesses emotion recognition Reading comprehension load Is this appropriate for your sample?

29 Other factors to Consider Computer/technical issues? Age-related perceptual decline? How well would tests work across different cultures? Test stakes? Test purpose? Motivational factors?

30 Question time

31 Back

32 Test Development Issue: Response Format

33 Response Format: What is the test-taker asked to do? Pick the best (MCQ) Pick the best and worst Rank them Rate each option Free response Allocate points among the options SCENARIO Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. OPTIONS (a) Nothing it will sort itself out soon enough. (b) Tell his family he feels left out. (c) Spend time listening and getting involved again. (d) Reflect that relationships can change over time. MSCEIT (and many tacit knowledge SJTs) STEU and STEM

34 Compare Response Formats MULTIPLE CHOICE Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. What action would be the most effective for Clayton? (a) Nothing it will sort itself out soon enough. (b) Tell his family he feels left out. (c) Spend time listening and getting involved again. (d) Reflect that relationships can change over time. Less internally consistent Strongerassociations with valued outcomes RATE-THE-EXTENT Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. Rate the effectiveness of the following actions 1 = not at all effective 5 = extremely effective? (a) Nothing it will sort itself out soon enough. (b) Tell his family he feels left out. (c) Spend time listening and getting involved again. (d) Reflect that relationships can change over time. More internally consistent Weakerassociations with valued outcomes

35 Study 1 (MacCann& Roberts, 2008) α Correlations STEU Stories Vocab SWLS Multiple choice (N=112) Rate-the-extent (N=91) sig Sig (correct for α) MacCann, C., & Roberts, R. D. (2008). New paradigms for assessing emotional intelligence: Theory and data. Emotion, 8,

36 Study 2 (N= 69): Short Form of STEM,20,10,00 -,10 -,20 -,30 -,40 Correlations of MCQ and Ratings-based formats with Criteria Anxiety Depression Stress Wellbeing LifeSat MCQ Rating RELIABILITY (18 items) Cronbach s alpha for ratings =.860 Cronbach s alpha for MCQ =.718

37 More on Rate-the-Extent Items: Ratings of Poor Emotion Management Strategies Rate-the extent formats involve an evaluation of poor emotion management strategies as well as good emotion management strategies MCQ just assesses whether you know the BEST thing to do Knowing how bad the worst options are may be important Elliot et al. (2011) interpersonal skills SJT:the poor options distinguished between experienced vsinexperienced teachers Is this true for the MSCEIT and other ratings-based tests? Are ratings of the poor responses more important for social/emotional functioning? Elliott, J., Stemler, S.E., Grigorenko, E.L., Sternberg, R.J. & Hoffman, N. (2011). The socially skilled teacher and the development of tacit knowledge. British Educational Research Journal 37,

38 Correlations with Criteria: MSCEIT Effective vs Ineffective Emotion Management Items Effective options (expert rating < 3/5) 13 items; M =.41 (SD =.07), α=.59 Ineffective options (expert rating > 3/5) 0,6 16 items; M =.35 (SD =.09), α=.67 N = 115 (Australian undergraduates) Link back to item 0,4 0,2 0,0-0,2 Significant differences -0,4 Ineffective Effective

39 Response Format: Conclusions Multiple choice may be preferred over ratings-based assessment Stronger evidence of validity If a rate-the-extent response format is used, assessments should include options representing poor choices Ratings of ineffective strategies for emotion management seem more important for predicting valued outcomes It might be that avoiding the worstbehaviours are more important than consistently enacting the best Replication is needed (ongoing)

40 Test Development Issue: Maximum versus typical performance instructions

41 Typical versus Maximum Performance Typical Performance Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. What WOULD YOU DOin this situation? (a) Nothing it will sort itself out soon enough. (b) Tell his family he feels left out. (c) Spend time listening and getting involved again. (d) Reflect that relationships can change over time. Behaviour (with reporting biases ) Higher reliability Stronger correlations with personality Lower scores, greater variability Can fake high (i.e., turn it into a knowledge test) Maximum Performance Clayton has been overseas for a long time and returns to visit his family. So much has changed that Clayton feels left out. What SHOULD Clayton do? (a) Nothing it will sort itself out soon enough. (b) Tell his family he feels left out. (c) Spend time listening and getting involved again. (d) Reflect that relationships can change over time. Knowledge Slightly lower reliability Stronger correlations with cognitive ability Higher scores, less variability Cannot fake high (need to know the answers) Freudenthaler, H. H., & Neubauer, A. C. (2007). Measuring emotional management abilities: Further evidence of the importance to distinguish between typical and maximum performance.personality and Individual Differences, 42(8),

42 Data on Typical vsmaximum Performance 0,5 0,4 0,3 0,2 0,1 0-0,1 STEM-Y (N= 138) O C E A ES g Maximum Typical Freudenthaler, H. H., & Neubauer, A. C. (2007). Measuring emotional management abilities: Further evidence of the importance to distinguish between typical and maximum performance.personality and Individual Differences, 42(8), ,6 0,5 0,4 0,3 0,2 0,1 0 0,5 0,4 0,3 0,2 0,1 0 Intrapersonal Abilities O C E A ES g Maximum Typical Interpersonal Abilities O C E A ES g Maximum Typical

Emotional Intelligence and Beyond: Looking Toward the Future. John D. Mayer

Emotional Intelligence and Beyond: Looking Toward the Future John D. Mayer Acknowledgements and Disclosure of Interests In Spain Mr. Emilio Botín President of the Fundación M. Botín Mr. Rafael Benjumea