History of JDM Linear Judgment Models

Similar documents
The Lens Model and Linear Models of Judgment

Psychology 466: Judgment & Decision Making

Introduction to Preference and Decision Making

The Psychology of Inductive Inference

Heuristics & Biases:

Sensory Memory, Short-Term Memory & Working Memory

Answer the questions on the handout labeled: Four Famous Reasoning Problems. Try not to remember what you may have read about these problems!

Introduction to Long-Term Memory

to Cues Present at Test

Representativeness Heuristic and Conjunction Errors. Risk Attitude and Framing Effects

The Case Against Deliberative Decision Making

Brook's Image Scanning Experiment & Neuropsychological Evidence for Spatial Rehearsal

Consolidation of Memories. Memory in the Real World

Cognition. Mid-term 1. Top topics for Mid Term 1. Heads up! Mid-term exam next week

Feature Integration Theory

Introduction to Categorization Theory

Chapter 1. Dysfunctional Behavioral Cycles

Reasoning with Uncertainty. Reasoning with Uncertainty. Bayes Rule. Often, we want to reason from observable information to unobservable information

AQA A Level Psychology. Topic Companion. Memory. Joseph Sparks & Helen Lakin

ISC- GRADE XI HUMANITIES ( ) PSYCHOLOGY. Chapter 2- Methods of Psychology

CAUTIONS ABOUT THE PRACTICE EXAM

Dear Participants in the Brunswik Society

Introduction to Attention and Theories of Selective Attention

Media Multitasking: How it s Changing You & Your Students. Clifford Nass Stanford University

EMOTIONAL INTELLIGENCE QUESTIONNAIRE

Why do Psychologists Perform Research?

Beattie Learning Disabilities Continued Part 2 - Transcript

FAQ: Heuristics, Biases, and Alternatives

Inside View Versus Outside View Irregularity Hypothesis Attribute Substitution

Emotional Intelligence and NLP for better project people Lysa

What is Psychology? Science versus Intuition 1 a. What is psychology? (from text) b. Note the revision from class:

Managerial Decision Making: Session 6

keep track of other information like warning discuss with your doctor, and numbers of signs for relapse, things you want to

Chapter 9: Factorial Designs

The Wellbeing Course. Resource: Mental Skills. The Wellbeing Course was written by Professor Nick Titov and Dr Blake Dear

Choose an approach for your research problem

Assignment 4: True or Quasi-Experiment

A Helping Model of Problem Solving

Rubin Adams IMPROVING ATTENTION AND FOCUS OF LEARNERS

Choosing Life: Empowerment, Action, Results! CLEAR Menu Sessions. Substance Use Risk 2: What Are My External Drug and Alcohol Triggers?

Chapter 2. Behavioral Variability and Research

Leadership Beyond Reason

Chapter 11 Nonexperimental Quantitative Research Steps in Nonexperimental Research

Memory Schemas, Source Monitoring & Eyewitness Memory

Stress Prevention in 6 Steps S T E P 3 A P P R A I S E : C O G N I T I V E R E S T R U C T U R I N G

Feeling. Thinking. My Result: My Result: My Result: My Result:

Doing High Quality Field Research. Kim Elsbach University of California, Davis

Pearson Education Limited Edinburgh Gate Harlow Essex CM20 2JE England and Associated Companies throughout the world

Step 2 Challenging negative thoughts "Weeding"

Motivation CURRENT MOTIVATION CONSTRUCTS

Utilizing Student Development Theory to Guide Advising Conversations

CSC2130: Empirical Research Methods for Software Engineering

When Your Partner s Actions Seem Selfish, Inconsiderate, Immature, Inappropriate, or Bad in Some Other Way

CHAPTER 2: PERCEPTION, SELF, AND COMMUNICATION

Empowered by Psychometrics The Fundamentals of Psychometrics. Jim Wollack University of Wisconsin Madison

Stat 13, Intro. to Statistical Methods for the Life and Health Sciences.

Empirical Knowledge: based on observations. Answer questions why, whom, how, and when.

Chapter 3 Tools for Practical Theorizing: Theoretical Maps and Ecosystem Maps

The Scientific Method

Tests/subtests that may capture this skill a,b. How it might look in school or in the home c Response inhibition

How Math (and Vaccines) Keep You Safe From the Flu

In this chapter we discuss validity issues for quantitative research and for qualitative research.

Intro to Cognitive Neuroscience. Working memory

Final Exam: PSYC 300. Multiple Choice Items (1 point each)

Pooling Subjective Confidence Intervals

3 CONCEPTUAL FOUNDATIONS OF STATISTICS

11/18/2013. Correlational Research. Correlational Designs. Why Use a Correlational Design? CORRELATIONAL RESEARCH STUDIES

Reliability and Validity

Statistics Mathematics 243

CAN T WE ALL JUST GET ALONG?

The Fallacy of Taking Random Supplements

Helping Your Asperger s Adult-Child to Eliminate Thinking Errors

Psychology Research Process

CHAPTER 3: SOCIAL PERCEPTION: UNDERSTANDING OTHER PEOPLE CHAPTER OVERVIEW

The Science of Psychology

Chapter 1 Data Types and Data Collection. Brian Habing Department of Statistics University of South Carolina. Outline

Unconscious Processes and Marketing. Joel Weinberger, Ph.D. Owner, Implicit Strategies Professor of Psychology, Adelphi University

Handout: Instructions for 1-page proposal (including a sample)

ACADEMIC APPLICATION:

Overview of Experimentation

Psychometrics, Measurement Validity & Data Collection

How Does Analysis of Competing Hypotheses (ACH) Improve Intelligence Analysis?

Online Identity and the Digital Self

Cognitive Science and Bayes. The main question. Several views. Introduction. The main question Several views Examples. The heuristics view

Critical Thinking Assessment at MCC. How are we doing?

Research Questions, Variables, and Hypotheses: Part 2. Review. Hypotheses RCS /7/04. What are research questions? What are variables?

Implicit Attitude. Brian A. Nosek. University of Virginia. Mahzarin R. Banaji. Harvard University

Psychological Approach to Comparative Education Aneela Farooq Afshan Nisar

What is Psychology? chapter 1

Goldsmith. Marshall. FEEDFORWARD Coaching For Your Future. Coaching For Your Future. MMIX ChartHouse Learning. All Rights Reserved. What Is It?

Over the years, philosophers, psychoanalysts, psychologists, psychiatrists,

Results & Statistics: Description and Correlation. I. Scales of Measurement A Review

Usually we answer these questions by talking about the talent of top performers.

Chapter 1 Review Questions

Take a tour through a fictional online fundraising campaign. And you ll be ready to put your campaign online and start fundraising

Behaviorism: An essential survival tool for practitioners in autism

How to Think Straight About Psychology

Nonjudgmentally and Cognitive Therapy

Short-Term and Working Memory. Outline. What is memory? Short-term memory Working memory Working memory and the brain. Chapter 5

Ingredients of Difficult Conversations

Transcription:

Finish: Begin: History of JDM Linear Judgment Models Psychology 466: Judgment & Decision Making Instructor: John Miyamoto 10/03/2017: Lecture 02-1 Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren t needed to view the slides. You can disable or delete the macros without any change to the presentation.

Outline Brief history of the psychology of decision making Capacity limitations in human cognitive processes Four linear judgment models Why are psychologists interested in linear judgment models? How can we make decisions based on a linear judgment model? Psych 466, Miyamoto, Aut 17 History of Psychology of Decision Making 2

Cognitive Approach to Judgment & Decision Making (JDM) Cognitive limitations limitations on human cognitive capacity affect judgment and decision making Heuristics and biases movement: 1970 1990 (approx.) Reactions to heuristics and biases movement Evolutionary psychology Ecological psychology Naturalistic decision making Bayesian models of psychological processes Emotion in decision processes The Standard Memory Model Psych 466, Miyamoto, Aut '17 3

Working Memory (WM) Has Severe Capacity Limitations Magic number 7 ± 2 George Miller s famous paper. WM can only hold a limited number of "chunks" of information. Remember the sequence of digits: 50, 85, 8, 2, 67, 14, 80, 66, 77, 65, 5, 18, 4, 7, 26,... Remember the sequence of words: TOP, FROG, SIGN, SLIP,HALL, BOWL, PICK, GRIN, LACK,... Psych 466, Miyamoto, Aut '17 Same Slide but with Additional Comment re the Role of Attention 4

Working Memory (WM) Has Severe Capacity Limitations Magic number 7 ± 2 George Miller s famous paper. WM can only hold a limited number of "chunks" of information. When information is complex, people are forced to simplify the reasoning process. Simplifications can lead to distortions. EXCEPTION: With experience, one can learn to integrate complex information into a small number of "chunks." Example: Expert chess players can reason about complicated chess problems. Example: Experienced drivers can understand traffic situations that are actually very complex. Chunking depends on automatic processes: Automatic processes low cognitive demands Controlled processes high cognitive demands Psych 466, Miyamoto, Aut '17 Diagram of the Standard Memory Model (from Hastie & Dawes) 5

Sensory Input Buffers Working Memory Hastie & Dawes Fig. 1.1 Central Executive Phonological Buffer Goal Stack Visuospatial Buffer Long-Term Memory The Standard Cognitive Model of Human Memory Psych 466, Miyamoto, Aut '17 Sensory Registers 6

Sensory Input Buffers Hastie & Dawes Fig. 1.1 Working Memory Central Executive Phonological Buffer Goal Stack Visuospatial Buffer Long-Term Memory Sensory registers retain the sensory information for very brief periods of time. Psych 466, Miyamoto, Aut '17 Working Memory 7

Sensory Input Buffers Hastie & Dawes Fig. 1.1 Working Memory Central Executive Phonological Buffer Goal Stack Visuospatial Buffer Long-Term Memory Working memory (WM) holds a limited amount of information. Thoughts are actively manipulated in WM. Capacity limit for unrelated pieces of information is about 7 ± 2 pieces. Information in WM lasts less than 20 seconds if it is not actively processed. Psych 466, Miyamoto, Aut '17 Long-Term Memory 8

Sensory Input Buffers Hastie & Dawes Fig. 1.1 Working Memory Central Executive Phonological Buffer Goal Stack Visuospatial Buffer Long-Term Memory Long-term memory (LTM) retains information over longer periods of time (days, months, even years). LTM interacts with WM. Information is retrieved from LTM to be processed in WM. Psych 466, Miyamoto, Aut '17 General Hypothesis of Cognitive Research 9

Sensory Input Buffers Hastie & Dawes Fig. 1.1 Working Memory Central Executive Phonological Buffer Goal Stack Visuospatial Buffer Long-Term Memory General Hypothesis of Cognitive Research Limitations in working memory impose limitations on human ability to engage in complex reasoning. Decision making requires complex reasoning. Psych 466, Miyamoto, Aut '17 Summary So Far: Where Are We in this Lecture? 10

Where Are We in the Lecture? Normative and prescriptive decision models require complex representations and processing. Cognitive limitations cause us to simplify decisions, and this can produce errors. Rational agent model ignores issue of cognitive limitations. NEXT: How to Deal with Cognitive Complexity Intuitive judgment versus statistical models Brunswik s Lens Model of Human Judgment Linear models applied to making better choices Applications to clinical judgment Intuitive Judgment vs Actuarial Judgment Psych 466, Miyamoto, Aut '17 11

Intuitive Judgment versus Acturial Judgment Intuitive judgment Combine complex information in your head Make decision based on gut feeling Actuarial judgment (a.k.a. statistical model or linear model) Base decisions on a statistical decision rule. Intellectual warfare between cognitive psychology and clinical psychology. (Especially in 1950's - 1970's). Statistical Models Outperform Human Judges Psych 466, Miyamoto, Aut '17 12

Examples of Judgment Problems We will only consider decisions to which intuitive judgment and actuarial judgment (statistical methods) both apply. -------------------------------------------------------------------------- E.g., Clinicians attempt to identify patients with progressive brain dysfunction. Data = intellectual test results Experienced clinicians achieved 58% correct detection of new cases. Statistical model achieved 83% correct detection of new cases. E.g., Bank loan officer must decide which loan applications are good risks and which are bad risks. E.g., College admissions committee must decide which high school applicants will do well at the college level. Psych 466, Miyamoto, Aut '17 Critique of Clinical Judgment What Is It? 13

Psych 466, Miyamoto, Aut '17 General Finding: Stat Models Outperform Human Judges 14 Critique of Clinical Judgment Clinical insight does it exist? Clinical judgment what are its strengths? Clinical judgment what are its weaknesses? Accusation: Belief in the efficacy of intuitive clinical judgment is a cognitive conceit.

General Finding: Stat Models Outperform Human Judges Statistical models almost always outperform the human judges on clearly defined decision tasks. Human cognitive processes are good at noticing particular pieces of information. Does my friend look happy? Sad? Stressed? Irritated? Is the patient nervous? Defensive? Exhibitionistic? Human cognitive processes are not good at integrating multiple pieces of information. Can I predict how my friend will feel about a surprise party? Can the clinician predict how the patient will progress after 4 months of therapy? Implications of this Topic Psych 466, Miyamoto, Aut '17 15

Implications of this Topic We can improve human decisions by stressing what humans are good at:... noticing what are important issues that are relevant to a decision;... evaluating how good or bad something might be on a specific dimension; while avoiding what we are not good at:... combining complex information in our heads. Know thyself Make better decisions Brunswik's Lens Model Psych 466, Miyamoto, Aut '17 16

The Lens Model of Egon Brunswik To-be-judged criterion = the thing you are trying to predict. Figure 3.1 of Hastie & Dawes. Judgment = the judge s prediction (you are the judge) Cues = things you can observe about the criterion The lens model is a conceptualization of the structure of typical judgment problems. Psych 466, Miyamoto, Aut '17 Examples of Jdmt Problems: Explain Criterion, Cues & Judgment 17

Psych 466, Miyamoto, Aut '17 Brief Digression: Definition of Holistic Judgment 18

Holistic Judgment Holistic judgment - judgment of a complex trait from multiple cues by means of a single intuitive judgment as opposed to a calculation based on a formula. Example of Holistic Judgment Psych 466, Miyamoto, Aut '17 19

Example of Holistic Judgment Suppose a clinician has to decide whether a patient is suicidal and should be hospitalized. He or she considers patient s appearance, what patient says, the patient s background and record, previous interactions, etc. Eventually, the clinician makes a decision based on an intuitive integration of all this information. ------------------------------------------------------------------------- Note: Holistic judgment may include stages in which the judge considers component features of the decision. What makes it a holistic judgment is that the ultimate evaluation is made by the judge through an intuitive integration of all of the information about the decision. Digression on the Meaning of Holism Psych 466, Miyamoto, Aut '17 20

Terminology: Holism and Analytical Strategies Holistic judgments are sometimes called global judgments to distinguish them from judgments that evaluate the separate cues. Holism is sometimes written as wholism. Holistic judgment strategies are contrasted with analytical judgment strategies. Analytical judgment strategy: Break the decision into component parts; use an explicit calculation to combine these parts. Claims Regarding Holistic & Analytical Judgment Strategies Psych 466, Miyamoto, Aut '17 21

Claims Regarding Holistic & Analytical Judgment Strategies Claim: An analytical judgment strategy will generally produce better predictions than a holistic judgment strategy. E.g., clinical judgments are more often correct if they are made analytically than if they are made holistically. E.g., your own predictions of what will happen in sports events, on the stock market, and predictions about social behavior will be more often correct if they are made analytically than if they are made holistically. Related Claim: An analytic judgment strategy is better than an intuitive judgment strategy because it provides a better way to combine complex information. Analytical versus Intuitive Judgment Strategy (cont.) Psych 466, Miyamoto, Aut '17 22

Do Analytical Judgment Strategies Exclude Human Intuition? NO! A human judge is still needed to...... decide what are the important issues in a decision;... help create an analysis of the decision structure;... judge how good or bad an outcome would be on a particular dimension, e.g., social impact of a decision, health impact of a decision, political impact of a decision, etc. Four Linear Judgment Models Psych 466, Miyamoto, Aut '17 23

Four Linear Judgment Models 1. Using multiple regression on objective data for which the true state is known. Apply this model to new data for which the true state is not known. 2. Using multiple regression on judgment data where the true state is not known. 3. SMART (Simple Multi-Attribute Rating Technique) 4. Unit Weighting Model -------------------------------------------- Except for Model 4 (unit weighting), the models do NOT describe everyday judgment. Some natural models are similar to these models. Using any of these models can improve human performance. Psych 466, Miyamoto, Aut '17 Reminder: Brunswik's Lens Model 24

Four Linear Judgment Models 1. Using multiple regression on objective data for which the true state is known. Apply this model to new data for which the true state is not known. 2. Using multiple regression on judgment data where the true state is not known. 3. SMART (Simple Multi-Attribute Rating Technique) 4. Unit Weighting Model -------------------------------------------- Assignment 1: Explain how to use a linear judgment procedure to choose 1 UW course to take. Minimize issues of major or minor requirements. Compare intuitive judgment and linear judgment models. On Assignment 1, you only need to explain 1 of 4 methods. Psych 466, Miyamoto, Aut '17 Reminder: Brunswik's Lens Model 25

Next: Explain These Four Methods on a Concrete Judgment Problem Concrete Problem: Predicting college performance (GPA) based on information in a college application. Predictions based on a linear model how to produce them Predictions based on a model of the judge how to produce them Predictions based on SMART method Predictions based on unit weights how to produce them Assignment 1: Pick one of these four methods. Describe this method and then discuss its strengths and weaknesses relative to other methods (including intuitive judgment. Continue the Intro to Baron's College Admission Problem Psych 466, Miyamoto, Aut '17 26

Judgment Data Used in Baron's Chapter 20 COL = college GPA. This is the criterion. This is what the judge wants to predict. SAT = SAT score; REC = judge's rating of the recommendation; ESS = judge's rating of the student's essay; GPA = high school GPA. These are the cues. Comment re Qualitative Variables Psych 466, Miyamoto, Aut '17 27

Baron's College GPA Judgment Problem Using the terminology of the lens model, the GPA judgment problem looks like this: Terminology Criterion (what we want to predict) Cues (these are things we know) Judgment (this is our prediction) Example COL = College GPA of high school student GPA = High school GPA SAT = SAT test scores REC = recommendations (converted to ratings) ESS = essay quality (converted to a rating) PRE = Estimate (guess) of student s future college GPA Table Showing Data & Variables for the College Admission Example Psych 466, Miyamoto, Aut '17 28

Qualitative Variables Must Be Converted To Quantitative Variables SAT, GPA are already quantitative. REC = strength of recommending letters is qualitative; convert to quant measure by having humans rate the letters for how positive they are. ESS = quality of applicant s essay is qualitative; convert to quant measure by having humans rate the essay for how good it is. Psych 466, Miyamoto, Aut '17 Four Methods for Predicting Future Cases 29

Four Ways to Compute a Statistical Prediction Model Method 1: Multiple regression applied to existing data. Called a proper linear model Next Method 2: Multiple regression applied to a judge s predictions. Called a model of the judge Method 3: SMART Method with "importance" weights. Called the SMART method or importance weighting method Method 4: Unit weighting model. Called the unit weighting model or unit weighting method Multiple Linear Regression Psych 466, Miyamoto, Aut '17 30

Method 1: Multiple Linear Regression Multiple linear regression is a statistical method for finding a formula that predicts the criterion from a set of data. COL is the criterion These are the cues PRE = statistical prediction (predicted college GPA) produced by Method 1 Prediction Equation from the Multiple Regression Psych 466, Miyamoto, Aut '17 31

Prediction Equation with 4 Predictor Variables COL = the criterion = college GPA = to-be-predicted quantity SAT, REC, ESS, GPA are the cues (predictor variables) Prediction Equation PRE = 0.000175 SAT + 0.092 REC + 0.217 ESS + 1.893 GPA 5.161 The regression weights for the prediction equation are underlined above: A statistics program can compute the regression weights based on the data in the table. Example of a Regression Equation Psych 466, Miyamoto, Aut '17 32

Prediction Equation with 4 Predictor Variables COL = the criterion = to-be-predicted quantity SAT, REC, ESS, GPA are the cues (predictor variables) Prediction Equation PRE = 0.000175 SAT + 0.092 REC + 0.217 ESS + 1.893 GPA 5.161 Example: If a high school student has SAT = 1200, REC = 3.7, ESS = 3.9, GPA = 3.2, then we predict PRE = 0.000175 (1200) + 0.092 (3.7) + 0.217 (3.9) + 1.893 (3.2) 5.161 = 2.2933 How to Use the Multiple Regression Model to Predict New Cases Psych 466, Miyamoto, Aut '17 33

How We Use the Linear Regression Model Step 1: Apply multiple regression calculation to data for which the value of the criterion (COL) is known. This produces the prediction equation. Step 2: Use the prediction equation to predict the criterion (COL) for new cases where the value of the criterion is NOT known. Linear Model Outperforms the Human Judge Psych 466, Miyamoto, Aut '17 34

Linear Model Outperforms Human Judges In Baron s college GPA example, the statistical model makes more accurate predictions of college GPA than do expert human judges. Same finding on many other examples (see Dawes, Faust & Meehl paper) Is this surprising? Is it interesting from the standpoint of psychology? Who cares? Or who might care? Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues. Whether or not the judge's introspections are veridical, the results of these studies show that a much simpler combination of the cues can produce better predictions than the processes used by human judges. Return to Outline of Models Psych 466, Miyamoto, Aut '17 35

Four Ways to Compute a Statistical Prediction Model Method 1: Multiple regression applied to existing data. Called a proper linear model Method 2: Multiple regression applied to a judge s predictions. Called a model of the judge Next Method 3: SMART Method with "importance" weights. Called the SMART method or importance weighting method Method 4: Unit weighting model. Called the unit weighting model or unit weighting method Model of the Judge Psych 466, Miyamoto, Aut '17 36

Not Available Not Available Method 2: Model of the Judge Researcher has available the scores for SAT, REC, ESS and GPA. The values of the criterion (COL) are NOT available. Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD (next slide). Same Slide Except JUD Column Added to Table Psych 466, Miyamoto, Aut '17 37

Not Available Not Available Method 2: Model of the Judge Researcher has available the scores for SAT, REC, ESS and GPA. The values of the criterion (COL) are NOT available. Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD." Regression Equation for the Model of the Judge Psych 466, Miyamoto, Aut '17 38

Not Available Not Available Method 2: Model of the Judge It is an accident that in this example, JUD and MUD are identical Compute a regression model that predicts JUD (Model of the judge or MUD). Example: Policy Capturing. Example: MUD = ( 5x10 18 ) SAT + 0.1 REC + 0.1 ESS + 2.0 GPA 4.8 Use the MUD model to predict college GPA (COL) for these cases or future cases. Pstch 466, Miyamoto, Aut '17 Discussion of Model of the Judge 39

Discussion of Case 2: Model of the Judge (MUD) Empirical findings for Case 2 are the same as for Case 1 MUD more accurate than the intuitive judgments of the judge. Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues. Whether or not this is really true, studies show that a much simpler combination of the cues produces better predictions. We don't need to know the value of the criterion (COL) in order to find a statistical formula (prediction equation) that can outperform the judge.. Why Does MUD Outperform the Judge? Psych 466, Miyamoto, Aut '17 40

Why Does MUD Outperform the Judge? The model of the judge (MUD) is a model of the judge s decisions, not of the criterion (reality). The MUD outperforms the judge because the statistical formula is consistent it treats each case by the same formula. A human judge has random variations in his or her judgment. Random variations increase the inaccuracy (error) of the judge's predictions. A human judge sees various special circumstances (anecdotal memories) that suggest that a specific case should be judged differently from the standard procedure. Psych 466, Miyamoto, Aut '17 Four Methods SMART Method is Next 41

Four Ways to Compute a Statistical Prediction Model Method 1: Multiple regression applied to existing data. Called a proper linear model Method 2: Multiple regression applied to a judge s predictions Called a model of the judge Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method Method 4: Unit weighting model Called the unit weighting model or unit weighting method Next SMART Method Psych 466, Miyamoto, Aut '17 42

Method 3: The SMART Method Simple Multi-Attribute Rating Technique (SMART) A.k.a. Importance Weighting Model This technique lets us rank order the individual cases (high school students). It does not produce a predicted GPA of each student. A rank order tells us who is predicted to do better or worse. This is enough to guide our selection of students. The SMART method can be modified to actually predict college GPA (COL), but we won t go into this complication because it is not needed in a choice situation. Step by Step Explanation of SMART Method Psych 466, Miyamoto, Aut '17 43

Method 3: The SMART Method Step 1: Convert the predictors, SAT, REC, ESS and GPA to z-scores. Step 2: The judge decides what is the relative importance of these variables, e.g., SAT is twice as important as ESS, GPA is 50% more important than SAT, etc. Step 3: Create a prediction equation that reflects the judge s judgment as to the relative importance of the predictor variables. Example of Prediction of MAUT Model = Predicted Score = 2 Z.sat + 3 Z.rec + 1 z.ess + 2 Z.gpa Psych 466, Miyamoto, Aut '17 Tables Showing Original Data, Z-Scores, and Prediction 44

Method 3: The SMART Method Z-Scores Table on the left shows the initial data. Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student. Predicted Score = 2 Z.sat + 3 Z.rec + 1 z.ess + 2 Z.gpa The Predicted column tells you who is predicted to do better or worse. It does not tell you the predicted GPA. Psych 466, Miyamoto, Aut '17 Example: Computing the Predicted Score for One Student 45

Method 3: The SMART Method Z-Scores Example of Prediction of MAUT Model = Predicted Score = 2 Z.sat + 3 Z.rec + 1 z.ess + 2 Z.gpa Example for 1 st student: 2(2.39) + 3(.32) + 1(.41) + 2(1.58) = 9.31 Comment on How to Convert Prediction to a Predicted College GPA Psych 466, Miyamoto, Aut '17 46

Method 3: The SMART Method Z-Scores To convert the Predicted column to an predicted college GPA, you need to furnish a guess as to the mean and variance of college GPA s (unnecessary for your assignment). Producing a predicted college GPA by this method requires a few math tricks. They are not worth discussing in Psych 466. Findings for the SMART Method Psych 466, Miyamoto, Aut '17 47

Findings for Method 3: SMART Method Findings for Method 3 are the same as for Method 2 (Model of the Judge) SMART method is more accurate than the intuitive judgments. The predicted ratings that are computed by the SMART method correlate better with actual results than do the judges intuitive predictions. More Discussion of the SMART Method Psych 466, Miyamoto, Aut '17 48

Discussion of Method 3: SMART Method The SMART method does not require that we know the value of the criterion (COL) for a set of known cases. We don't need a complicated calculation of the optimal regression weights (Methods 1 & 2 require this calculation). The SMART Method is better than intuitive judgment because. It approximately captures the relative importance of the predictor variables, and. It is consistent it is not bothered by shifts of attention, fatigue, distractions, anecdotal memories, etc. Four Methods Next We Discuss Unit Weighting Model Psych 466, Miyamoto, Aut '17 49

Four Ways to Compute a Statistical Prediction Model Method 1: Multiple regression applied to existing data. Called a proper linear model Method 2: Multiple regression applied to a judge s predictions Called a model of the judge Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method Method 4: Unit weighting model Called the unit weighting model or unit weighting method Next Unit Weighting Method Psych 466, Miyamoto, Aut '17 50

Reminder: What Are the Weights? Method 1: Proper Linear Model PRE = 0.000175 SAT + 0.092 REC + 0.217 ESS + 1.893 GPA 5.161 Method 2: Prediction Equation for the Judge s Predictions MUD = ( 5x10 18 ) SAT + 0.1 REC + 0.1 ESS + 2.0 GPA 4.8 Method 3: Predicted Scores Based on Importance Weights: Predicted Score = 2 Z.sat + 3 Z.rec + 1 z.ess + 2 Z.gpa Method 4: Unit Weighting Model: Predicted Score = 1 Z.sat + 1 Z.rec + 1 z.ess + 1 Z.gpa The weights are the numbers that are assigned to the predictor variables. Further Explanation of the Unit Weighting Model Psych 466, Miyamoto, Aut '17 51

Method 4: Unit Weighting Model Z-Scores Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student. Example for Case 1: 1(2.39) + 1(.32) + 1(.41) + 1(1.58) = 4.70 Findings for the Unit Weighting Model Psych 466, Miyamoto, Aut '17 52

Findings for Method 4: Unit Weighting Model Unit weighting model more accurate than the intuitive judgments of the judge. Judges think that they make their judgments by means of a complex, interactive, nonlinear evaluation of the cues. Whether or not this is really true, studies show that a much simpler combination of the cues produces better predictions. We don't even need to calculate optimal regression weights. All we need to know is which cues are positively related to the criterion and which are negatively related to the criterion. Discussion of the Unit Weighting Model (cont.) Psych 466, Miyamoto, Aut '17 53

Discussion of Method 4: Unit Weighting Model (cont.) Unit weighting model outperforms human judge because it is consistent - it treats each case by the same formula. A human judge has all sorts of random variations in his or her judgment. These random variations simply increase the inaccuracy (error) of the judge's predictions. A human judge may be influenced by anecdotal memories. The unit weighting model is not influenced by these memories. Comment: Results for unit weighting model show that the critical weakness of the human judge is inconsistency, not the inability to produce optimal weights. Table: Pros and Cons of Prediction Methods Psych 466, Miyamoto, Aut '17 54

Psych 466, Miyamoto, Aut '17 Repeat this Table Without the Red Rectangles 55 Only if you want to make a quantitative prediction

Psych 466, Miyamoto, Aut '17 Conclusions 56

Tuesday, 3 October, 2017: The Lecture Ended Here Psych 466,, Miyamoto, Aut '17 57

What We Have Learned from the Study of Linear Judgment Models Human judges typically believe that they use complex judgment processes to make judgments from complex cues. (This may be true.) The part of the human judgment process that validly predicts the criterion is well modeled by a simple linear model. We don't need to use optimal regression models to outperform human judges. We don't need to know the value of the criterion in order to create a model that outperforms human judges. Evidence: Model of the judge, unit weighting and importance weighting models outperform the human judge. Note: These methods work only because the human judge has some valid knowledge of the relationship between the cues and the criterion. Psych 466, Miyamoto, Aut '17 Sample Size Affects the Predictive Accuracy of Multiple Regression & MUD 58

Sample Size Affects the Accuracy of Multiple Regression Model and Model of the Judge This point is emphasized by Gigerenzer: If you are using a multiple regression model or the model of the judge, the predicted accuracy of the model is bad if the sample size is too small. If the sample size is small, then the regression weights will tend to be inaccurate. (Technically, the variance of the regression weights greatly increases as the sample size gets smaller.) If the sample size is small, the unit weighting model can be more accurate than multiple regression model or the model of the judge. (This point is emphasized by Gigerenzer.) Psych 466, Miyamoto, Aut '17 Why Do Statistical Model Outperform Humans? 59

Why Do Statistical Models Outperform Human Judges? Human judgment is affected by internal random variation; statistical model is not. Human judgment is affected by vivid individual cases (anecdotes); statistical model is not. Speculation: Human judge tries to fit information into a story; statistical model ignores story; it just adds up the evidence. But is the human preference for stories bad? Psych 466, Miyamoto, Aut '17 General Discussion of Linear Judgment Models - END 60

General Discussion of Linear Models Why aren t linear judgment models used more widely in practical decision making? College or graduate admissions NIH or NSF grant review committees Political decisions like where to locate a prison; where to locate a homeless shelter; The Denver bullet study Are linear judgment methods dehumanizing, e.g., when choosing who will be admitted to a college? Psych 466, Miyamoto, Aut '17 END 61

Set Up for Instructor Turn off your cell phone. Close web browsers if they are not needed. Classroom Support Services (CSS), 35 Kane Hall, 206-543-9900 If the display is odd, try setting your resolution to 1024 by 768 Run Powerpoint. For most reliable start up: Start laptop & projector before connecting them together If necessary, reboot the laptop Psych 466, Miyamoto, Aut 17 62