Brunswik-Symmetry: A key concept for successful assessment tin education and elsewhere. Werner W. Wittmann University of Mannheim, Germany

Size: px

Start display at page:

Download "Brunswik-Symmetry: A key concept for successful assessment tin education and elsewhere. Werner W. Wittmann University of Mannheim, Germany"

Emma Warren
6 years ago
Views:

1 Brunswik-Symmetry: A key concept for successful assessment tin education and elsewhere Werner W. Wittmann University of Mannheim, Germany Session III: Diagnostics in Everyday Cognition Chair: Patrick Kyllonen (Educational Testing Service) 4th Spearman Conference (ETS) Philadelphia, Oct , 2004

2 Outline Part 1: Conceptualizations which those who know me probably already know Part 2: Consequences and data which those you know me probably don t know

3 Symmetry a golden key concept in science In all major scientific disciplines principles of symmetry play a major role in developing successful theories. The evidence goes from Benjamin Franklin and Michael Faraday to Richard Feynman and Murray Gell-Mann. What about psychology and education?

4 THE LENS MODEL: Clinical prediction paradigm schematized by Brunswik s lens model (After Hammond, Hursch, and Todd, 1964) Validity (achievement), r yc y j X 1 r x1 y c r x1 y j r x2 y c X 2 r x2 y j Criterion score y c r x 3 y c r x3 x 3 y j y j Clinician s prediction r x4 y c X 3 r x4 y j Empirical validity of cues, r xi y c X y = b x + b x + b x + b x 4 c c 1 1 c 2 2 c 3 3 c 4 4 Input data (cues) Cue utilization by clinician, r xi y j y = b x + b x + b x + j j 1 1 j 2 2 j 3 3 b x j 4 4

5 Egon Brunswik (Photo courtesy of Department of Psychology, University of California, Berkeley) Thank you Egon and congratulations to your 101 st anniversary

6 The true Brunswik-symmetrical latent structure of nature Pr1 Cr1 PR A Pr2 Cr2 CR A Pr3 Cr3 PR g CR g Pr4 Cr4 PR B Pr5 Cr5 CR B Pr6 Cr6 r true = 1 r true = 1 r true = 1

7 Tucker s (1964) lens model equation r PR,CR = G PR,CR R PR R CR + C PR,CR ( 2 ) ( 2 1 R 1 R ) PR CR Linear part Non linear part plus random error

8 The Brunswik-lens-equation equation for relating predictors (PR) to criteria (CR) true effect 678 observed PR CR true rpr,cr = S rtt rtt GPR,CR 123 RPR RCR e Selection effects due to restriction (enhancement) of range 1 Danger to overestimate 1 Danger to underestimate Psychometric reliability of predictor and criterion 2 Dangers to underestimate Construct reliability of predictor and criterion 2 Dangers to underestimate (lack of symmetry) Sampling error 1 Danger to overestimate (positive error) 1 Danger to underestimate (negative error) There 6 dangers to underestimate There 6 dangers to underestimate against 2 dangers to overestimate A true effect size!

9 The Five Data-Box Conceptualization PRE-Intervention Intervention, program or treatment Post intervention area of criteria and goals Northwestern Path ETR-Box Area of Stakeholder interests PR-Box r PR, CR EVA-Box CR-Box Southwestern Path NTR-Box

10 On an ongoing basis, the What Works Clearinghouse (WWC) gathers studies of the effectiveness of educational interventions (programs, products, practices, and policies). We review the studies that have the strongest design, and report on the strengths and weaknesses of those studies against the WWC Evidence Standards so that you know what the best scientific evidence has to say.

11 The beauty of Brunswik-Symmetry ETR-Box PR-Box CR-Box NTR-Box ETR - Experimental Treatment Box PR - Predictor Box CR - Criteria Box NTR - Nonexperimental Treatment Box

12 FOUR VARIANTS OF ASYMMETRY

13 Full asymmetry, the case of nothing works! All correlations between predictors and criteria are zero! Hierarchy of predictors Hierarchy of criteria

14 Asymmetry due to a broad higher level predictor and a narrower lower level l criterion i Hierarchy of predictors Hierarchy of criteria

15 Asymmetry due to a narrower lower level predictor and a broad higher level criterion Hierarchy of predictors Hierarchy of criteria

16 The hybrid case of asymmetry, mismatch at the same level generality! Hierarchy of predictors Hierarchy of criteria

17 The danger of asymmetry: Narrowing the predictor box Predictor-Box Criterion I School/College/ University Criterion II Real Business Life V PR V CR g PR g PR GPA CR NCR N PR Super- visor g CR

18 The danger of asymmetry in validation strategies Predictor-Box Criterion I School/College/ University Criterion II Real Business Life? CR N PR N g PR N CR g CR V PR GPA CR V CR F PR F CR

19 The danger of asymmetry in validation strategies Predictor-Box Criterion I School/College/ University Criterion II Real Business Life? CR N PR N CR g PR V PR GPA CR V CR g CR F PR F CR

20 Demands on the workforce CP SNOW s distinction of the two cultures Buzz Hunt s question: Will we be smart enough? Camilla Benbow and David Lubinski s focus on tilted profiles in aptitude and achievement

21 Tilted Profiles Upper third VERBAL QUANT Middle third Lower third

Tilted Profiles 800 800 Upper third Middle

22 Tilted Profiles Upper third Middle third VERBAL QUANT Lower third

23 Tilted Profiles Upper third Middle third Lower third VERBAL QUANT

24 Tilted Profiles There are many women with quant tilted profiles but proportionally less than men QUANT VERBAL

25 Tilted Profiles Upper third QUANT VERBAL Middle third Lower third

26 Tilted Profiles Upper third Middle third QUANT VERBAL Lower third

27 Tilted Profiles Upper third Middle third Lower third QUANT VERBAL

28 Tilted Profiles Yes, there are men with verbal tilted profiles but proportionally less than women VERBAL QUANT

29 Tilted Profiles Even profiles are very attractive ti VERBAL QUANT

30 Tilted Profiles QUANT VERBAL University of...

31 Tilted Profiles VERBAL QUANT University of...

32 Tilted Profiles VERBAL QUANT University of...

33 International benchmarks: Evidence from the PISA-OECD database In 2000 representative samples of fifteen years old students were assessed in reading, math and science in 32 OECD countries Today these cohorts are in colleges, universities,vocational schools or in the workforce We computed level and shape of their profiles in verbal and quant achievements in PISA The first unrotated principal component of reading, math and science tests represents the level differences and the second one contrasted reading and math and maps the shape (tiltedness) of the profiles

34 Quant and verbal tilted countries PISA_OECD study Quant tilted Verbal tilted Korea Japan Switzerland Denmark Iceland Norway Luxembourg New Zealand Hungary Austria Sweden Czech Republ UK Germany Australia Poland Belgium Canada USA Greece France Portugal Finland Spain Mexico Ireland Italy 0.717

35 Quant tilted Tilted profiles by country and gender Verbal tilted SWITZERLAND male JAPAN male NETHERLANDS male RUSSIA female KOREA male IRELAND male AUSTRIA male NEW ZEALAND female LATVIA male LUXEMBOURG female LUXEMBOURG male DENMARK female BELGIUM male BELGIUM female CZECH REPUBL male DENMARK male RUSSIA male NEW ZEALAND male GERMANY male SWEDEN male FRANCE male POLAND male NORWAY male ICELAND male FINLAND male AUSTRALIA male UK male HUNGARY male CANADA male SWITZERLAND female NETHERLANDS female PORTUGAL male SPAIN male BRAZIL male GREECE male USA male JAPAN female KOREA female MEXICO male ITALY male AUSTRIA female SWEDEN female GERMANY female ICELAND female LATVIA female UK female AUSTRALIA female CZECH REPUBL female BRAZIL female HUNGARY female POLAND female FRANCE female CANADA female USA female MEXICO female FINLAND female PORTUGAL female NORWAY female SPAIN female GREECE female ITALY female IRELAND female 0.683

36 Pisa-profiles: Level and shape group percentages The following results are for: 32 OECD Countries Percents of total count all PISA_OECD countries Tiltedness/shape (rows) by level (columns) of PISA-profiles lower middle upper Total N third third third Verbal ,648 Even ,649 Quant , Total N 11,648 11,649 11,649 34,946

37 Pisa-profiles: Level and shape group percentages The following results are for: United States Percents of total count Tiltedness (rows) by level (columns) of PISA-profiles lower middle upper Total N third third third Verbal Even Quant Total N females & males

38 Pisa-profiles: Level and shape group percentages The following results are for: United States Percents of total count Tiltedness (rows) by level of profile (columns) m= male; f = female low middle high Total N Verbal f m Even f m Quant f m Total f m N f m

39 Gender enrollment proportions in higher h education

40 Attempts to bring real life into the lab to predict and explain performance Computer based complex decision scenarios are an interesting possibility to simulate real life demands in the lab My Mannheim research group on working memory, intelligence and complex problem solving performance focussed on the relationships between these constructs We used mainly three different scenarios called: -Tailorshop -Powerplant -Learn

41 Testing Ackerman s PPIK-theory WMC95_study Chi sq.=28.51 P=0.87 CFI=1.00 RMSEA=0.00

42 The Mannheim (1997) study The importance of intelligence as process R =.715 R 2 =.511 adj. R 2 =.479 N = 135 Prediction and explanation of performance from the group-factor level WMC-g = General factor of all working memory tasks; BIS = Berlin intelligence structure; KNOW-g = total knowledge; PL3 = total computer games performance; WMC-SPAT = Spatial working memory factor; WMC-NV = Verbal-Numerical working memory factor; WMC-SUP = Processing speed working memory factor; K = reasoning; M = short-term memory; B = speed; E = creativity. EQS Summary Statistics Method: ML Model CHI-Square df = 22 p-value BBNFI BBNNFI CFI

43 Higher symmetry at a lower level of generality

44 Summary Principles of symmetry give us a wake-up call, how much we neglect the criterion problem The potential to fool ourselves and others in disregarding Brunswik-symmetry and the related principles of correspondance in levels of generality (Fishbein & Aizen) is enormous Will we be smart enough to react, adapt and fit the levels and the shape of those educated by us to the demands of the future workplaces? As always in this process there will be winners and loosers Hopefully it is your and my team which are among the winners!

508 the number of suicide deaths in deaths per 100,000 people was the suicide rate in Suicide deaths in 2013 by gender

508 the number of suicide deaths in deaths per 100,000 people was the suicide rate in Suicide deaths in 2013 by gender An overview of suicide statistics This document summarises information about suicide deaths in New Zealand covering up to 13. It does not attempt to explain causes of suicidal behaviour or causes of changes