Brunswik-Symmetry: A key concept for successful assessment tin education and elsewhere Werner W. Wittmann University of Mannheim, Germany Session III: Diagnostics in Everyday Cognition Chair: Patrick Kyllonen (Educational Testing Service) 4th Spearman Conference (ETS) Philadelphia, Oct. 20-23, 2004
Outline Part 1: Conceptualizations which those who know me probably already know Part 2: Consequences and data which those you know me probably don t know
Symmetry a golden key concept in science In all major scientific disciplines principles of symmetry play a major role in developing successful theories. The evidence goes from Benjamin Franklin and Michael Faraday to Richard Feynman and Murray Gell-Mann. What about psychology and education?
THE LENS MODEL: Clinical prediction paradigm schematized by Brunswik s lens model (After Hammond, Hursch, and Todd, 1964) Validity (achievement), r yc y j X 1 r x1 y c r x1 y j r x2 y c X 2 r x2 y j Criterion score y c r x 3 y c r x3 x 3 y j y j Clinician s prediction r x4 y c X 3 r x4 y j Empirical validity of cues, r xi y c X y = b x + b x + b x + b x 4 c c 1 1 c 2 2 c 3 3 c 4 4 Input data (cues) Cue utilization by clinician, r xi y j y = b x + b x + b x + j j 1 1 j 2 2 j 3 3 b x j 4 4
Egon Brunswik 1903-1955 (Photo courtesy of Department of Psychology, University of California, Berkeley) Thank you Egon and congratulations to your 101 st anniversary
The true Brunswik-symmetrical latent structure of nature Pr1 Cr1 PR A Pr2 Cr2 CR A Pr3 Cr3 PR g CR g Pr4 Cr4 PR B Pr5 Cr5 CR B Pr6 Cr6 r true = 1 r true = 1 r true = 1
Tucker s (1964) lens model equation r PR,CR = G PR,CR R PR R CR + C PR,CR ( 2 ) ( 2 1 R 1 R ) PR CR Linear part Non linear part plus random error
The Brunswik-lens-equation equation for relating predictors (PR) to criteria (CR) true effect 678 observed PR CR true rpr,cr = S rtt rtt GPR,CR 123 RPR RCR 14243 + e Selection effects due to restriction (enhancement) of range 1 Danger to overestimate 1 Danger to underestimate Psychometric reliability of predictor and criterion 2 Dangers to underestimate Construct reliability of predictor and criterion 2 Dangers to underestimate (lack of symmetry) Sampling error 1 Danger to overestimate (positive error) 1 Danger to underestimate (negative error) There 6 dangers to underestimate There 6 dangers to underestimate against 2 dangers to overestimate A true effect size!
The Five Data-Box Conceptualization PRE-Intervention Intervention, program or treatment Post intervention area of criteria and goals Northwestern Path ETR-Box Area of Stakeholder interests PR-Box r PR, CR EVA-Box CR-Box Southwestern Path NTR-Box
On an ongoing basis, the What Works Clearinghouse (WWC) gathers studies of the effectiveness of educational interventions (programs, products, practices, and policies). We review the studies that have the strongest design, and report on the strengths and weaknesses of those studies against the WWC Evidence Standards so that you know what the best scientific evidence has to say.
The beauty of Brunswik-Symmetry ETR-Box PR-Box CR-Box NTR-Box ETR - Experimental Treatment Box PR - Predictor Box CR - Criteria Box NTR - Nonexperimental Treatment Box
FOUR VARIANTS OF ASYMMETRY
Full asymmetry, the case of nothing works! All correlations between predictors and criteria are zero! Hierarchy of predictors Hierarchy of criteria
Asymmetry due to a broad higher level predictor and a narrower lower level l criterion i Hierarchy of predictors Hierarchy of criteria
Asymmetry due to a narrower lower level predictor and a broad higher level criterion Hierarchy of predictors Hierarchy of criteria
The hybrid case of asymmetry, mismatch at the same level generality! Hierarchy of predictors Hierarchy of criteria
The danger of asymmetry: Narrowing the predictor box Predictor-Box Criterion I School/College/ University Criterion II Real Business Life V PR V CR g PR g PR GPA CR NCR N PR Super- visor g CR
The danger of asymmetry in validation strategies Predictor-Box Criterion I School/College/ University Criterion II Real Business Life? CR N PR N g PR N CR g CR V PR GPA CR V CR F PR F CR
The danger of asymmetry in validation strategies Predictor-Box Criterion I School/College/ University Criterion II Real Business Life? CR N PR N CR g PR V PR GPA CR V CR g CR F PR F CR
Demands on the workforce CP SNOW s distinction of the two cultures Buzz Hunt s question: Will we be smart enough? Camilla Benbow and David Lubinski s focus on tilted profiles in aptitude and achievement
Tilted Profiles 800 800 Upper third VERBAL QUANT Middle third 500 500 Lower third 200 200
Tilted Profiles 800 800 Upper third Middle third 500 500 VERBAL QUANT Lower third 200 200
Tilted Profiles 800 800 Upper third Middle third 500 500 Lower third VERBAL QUANT 200 200
Tilted Profiles There are many women with quant tilted profiles but proportionally less than men 800 800 QUANT 500 500 VERBAL 200 200
Tilted Profiles 800 800 Upper third QUANT VERBAL Middle third 500 500 Lower third 200 200
Tilted Profiles 800 800 Upper third Middle third 500 500 QUANT VERBAL Lower third 200 200
Tilted Profiles 800 800 Upper third Middle third 500 500 Lower third QUANT VERBAL 200 200
Tilted Profiles Yes, there are men with verbal tilted profiles but proportionally less than women 800 800 VERBAL 500 500 QUANT 200 200
Tilted Profiles Even profiles are very attractive ti 800 800 VERBAL QUANT 500 500 200 200
Tilted Profiles 800 800 QUANT 500 500 VERBAL 200 200 University of...
Tilted Profiles 800 800 VERBAL 500 500 QUANT 200 200 University of...
Tilted Profiles 800 800 500 500 VERBAL QUANT 200 200 University of...
International benchmarks: Evidence from the PISA-OECD database In 2000 representative samples of fifteen years old students were assessed in reading, math and science in 32 OECD countries Today these cohorts are in colleges, universities,vocational schools or in the workforce We computed level and shape of their profiles in verbal and quant achievements in PISA The first unrotated principal component of reading, math and science tests represents the level differences and the second one contrasted reading and math and maps the shape (tiltedness) of the profiles
Quant and verbal tilted countries PISA_OECD study Quant tilted Verbal tilted Korea -0.677 Japan -0.629 Switzerland -0.613 Denmark -0.357 Iceland -0.310 Norway -0.288 Luxembourg -0.255 New Zealand -0.168 Hungary -0.113 Austria -0.064 064 Sweden -0.062 Czech Republ -0.025 UK -0.014 Germany -0.007 Australia 0.028 Poland 0.040 Belgium 0.046 Canada 0.061 USA 0.142 Greece 0.160 France 0.187 Portugal 0.261 Finland 0.282 Spain 0.338 Mexico 0.503 Ireland 0.567 Italy 0.717
Quant tilted Tilted profiles by country and gender Verbal tilted SWITZERLAND male -0.809 JAPAN male -0.620 NETHERLANDS male -0.598 RUSSIA female 0.110 KOREA male -0.574 IRELAND male 0.117 AUSTRIA male -0.532 NEW ZEALAND female 0.153 LATVIA male -0.474 LUXEMBOURG female 0.175 LUXEMBOURG male -0.440 DENMARK female 0.199 BELGIUM male -0.407 BELGIUM female 0.202 CZECH REPUBL male -0.406 06 DENMARK male -0.398 RUSSIA male -0.385 NEW ZEALAND male -0.384 GERMANY male -0.379 SWEDEN male -0.347 FRANCE male -0.336 POLAND male -0.318 NORWAY male -0.316 ICELAND male -0.311 FINLAND male -0.290 AUSTRALIA male -0.271 UK male -0.265 HUNGARY male -0.264 CANADA male -0.205 SWITZERLAND female -0.122 NETHERLANDS female -0.108 PORTUGAL male -0.107 SPAIN male -0.105 BRAZIL male -0.090 GREECE male -0.082 USA male -0.066 JAPAN female -0.055 KOREA female -0.051 MEXICO male -0.045 ITALY male -0.026 AUSTRIA female 0.221 SWEDEN female 0.230 GERMANY female 0.231 ICELAND female 0.245 LATVIA female 0.266 UK female 0.277 AUSTRALIA female 0.300 CZECH REPUBL female 0.303 BRAZIL female 0.321 HUNGARY female 0.331 POLAND female 0.353 FRANCE female 0.377 CANADA female 0.378 USA female 0.403 MEXICO female 0.422 FINLAND female 0.434 PORTUGAL female 0.448 NORWAY female 0.466 SPAIN female 0.521 GREECE female 0.602 ITALY female 0.641 IRELAND female 0.683
Pisa-profiles: Level and shape group percentages The following results are for: 32 OECD Countries Percents of total count all PISA_OECD countries Tiltedness/shape (rows) by level (columns) of PISA-profiles lower middle upper Total N third third third Verbal 11.214 11.383 10.734 33.33111,648 Even 10.428 10.997 11.910 33.33411,649 Quant 11.689 10.954 10.691 33.33411,64933411 Total 33.331 33.334 33.334 100.000 N 11,648 11,649 11,649 34,946
Pisa-profiles: Level and shape group percentages The following results are for: United States Percents of total count Tiltedness (rows) by level (columns) of PISA-profiles lower middle upper Total N third third third Verbal 14.510 14.148 12.938 41.596 344 Even 13.906 10.641 9.674 34.220 283 Quant 11.487 6.651651 6.046 24.184 200 Total 39.903 31.439 28.658 100.000 N 330 260 237 827 females & males
Pisa-profiles: Level and shape group percentages The following results are for: United States Percents of total count Tiltedness (rows) by level of profile (columns) m= male; f = female low middle high Total N Verbal f 18.225 17.746 17.026 52.998 221 m 10.732 10.488 8.780 30.000 123 Even f 11.751 10.312 8.873 30.935 129 m 16.098 10.976 10.488 37.561 154 Quant f 6954 6.954 4.796 4317 4.317 16.067 67 m 16.098 8.537 7.805 32.439 133 Total f 39.903 31.439 28.658 100.000 m 42.927 30.000 27.073 100.000 N f 154 137 126 417 m 176 123 111 410
Gender enrollment proportions in higher h education
Attempts to bring real life into the lab to predict and explain performance Computer based complex decision scenarios are an interesting possibility to simulate real life demands in the lab My Mannheim research group on working memory, intelligence and complex problem solving performance focussed on the relationships between these constructs We used mainly three different scenarios called: -Tailorshop -Powerplant -Learn
Testing Ackerman s PPIK-theory WMC95_study Chi sq.=28.51 P=0.87 CFI=1.00 RMSEA=0.00
The Mannheim (1997) study The importance of intelligence as process R =.715 R 2 =.511 adj. R 2 =.479 N = 135 Prediction and explanation of performance from the group-factor level WMC-g = General factor of all working memory tasks; BIS = Berlin intelligence structure; KNOW-g = total knowledge; PL3 = total computer games performance; WMC-SPAT = Spatial working memory factor; WMC-NV = Verbal-Numerical working memory factor; WMC-SUP = Processing speed working memory factor; K = reasoning; M = short-term memory; B = speed; E = creativity. EQS Summary Statistics Method: ML Model CHI-Square df = 22 p-value BBNFI BBNNFI CFI 21.93 0.4642 0 928 0.928 1.000 1.000
Higher symmetry at a lower level of generality
Summary Principles of symmetry give us a wake-up call, how much we neglect the criterion problem The potential to fool ourselves and others in disregarding Brunswik-symmetry and the related principles of correspondance in levels of generality (Fishbein & Aizen) is enormous Will we be smart enough to react, adapt and fit the levels and the shape of those educated by us to the demands of the future workplaces? As always in this process there will be winners and loosers Hopefully it is your and my team which are among the winners!