MEASUREMENT THEORY Measurement Theory How do we measure things that aren t material? Affect, behavior, cognition Thoughts, feelings, personality, intelligence, etc? Measurement theory: Turning the into something we can measure Latent Variable that cannot be directly measured and so must be inferred by something else Latent Variables Usually s that are relatively stable, not states Represented graphically as an oval Latent Variable Inferred from measured s, represented graphically as boxes Variable 1
Example = Latent Variable No direct measure of how highly extraverted someone is So, we need to find ways to extraversion Example How sociable are you? Behavioral history: How many parties did person attended last month? Behavioral observation: Observe during getting to know you activity Example How sociable are you? How long does it take you to be comfortable around new people? Are you usually the life of the party? 2
Example Latent s often measured with questionnaires, because surveys can easily assess emotions, cognitions and past behaviors Latent Variable How sociable are you? How long does it take you to be comfortable around new people? Are you usually the life of the party? Nested Latent Variables Latent Variables can stack or become nested when a has multiple dimensions E.g., : Highly Social Need for Excitement Low Social Anxiety Nested Variables Model Highly Social Need for Excitement Low Social Anxiety Model 3
Steps in Measuring a Variable 1. Figure out its latent structure What factors or components capture the entire latent? High Sociability Need for Excitement High Impulsivity Steps in Measuring a Variable 2. Construct to assess factors What measures could capture the entire factor? Behavioral Mixed Self-report Other-report Methods Observation Questionnaire Is talkative Is reserved Tends to be quiet High Sociability Is outgoing Steps in Measuring a Variable 3. Item Creation Construct multiple items to assess each factor that underlies the latent E.g., extraversion scales could include items measuring sociability, impulsivity, need for excitement Make some of the items E.g., I am very outgoing vs. I am quite shy Participants tend to show positivity bias (answering somewhat positively to every question) Reverse-scored items cancel out this bias 4
8/15/17 Steps in Measuring a Variable 3. Item Creation All questions should have the same response set E.g., How much do you agree with each statement? 1 = strongly disagree 2 = somewhat disagree 3 = somewhere in the middle 4 = somewhat agree 5 = strongly agree Steps in Measuring a Variable 4. Develop a Response Set Scales: Scale from 1 X E.g., 1 = strongly disagree, 2 = somewhat disagree, 5 7 point scales work best Gives participants enough options without over-whelming them Infinite Scales E.g., circle your answer: Strongly disagree Strongly agree Steps in Measuring a Variable 5. Pilot Testing To assess whether you have a good scale, you need to test it s : How consistent is the scale in measuring the latent? : How accurate is the scale in measuring the latent? 6. Use the scale in research 5
does the measure assess traits/ behavior consistently? E.g., Someone takes intelligence tests multiple times does the measure assess traits/ behavior consistently? Distribution of a scale with reliability Wide variance of scores does the measure assess traits/ behavior consistently? E.g., Someone takes intelligence tests multiple times 6
does the measure assess traits/ behavior consistently? Distribution of a scale with reliability Highly consistent scores does the measure assess traits/ behavior consistently? E.g., Someone takes intelligence tests multiple times More Reliable Number of Tests w/score 45 40 35 30 25 20 15 10 5 0 IQ Score Number of Tests w/score 45 40 35 30 25 20 15 10 5 0 IQ Score Every score on a measure is composed of : What the 45 Signal score actually should be 40 (in a perfect world) 35 Called the signal : Random variance Called noise Number of Tests w/score 30 25 20 15 10 5 0 Noise IQ Score 7
Every score on a measure is composed of Signal A reliable measure 45 minimizes the noise 40 Minimizing errors due to bad Measuring exactly the same thing every time Measuring the desired Number of Tests w/score 35 30 25 20 15 10 5 0 Noise IQ Score Methods of Testing = Is a measure consistent? Test the same 2 or more times with the same individuals A participant s 1 st test score should correlate highly with their 2 nd test score Have to space measures apart to avoid effects Methods of Testing = Is a measure consistent? For measures that are composed of multiple questions If questions measure the same thing, they should be highly correlated Split-half method half the items are correlated with the other half each item is correlated with all other items and then an average correlation is calculated If average r =.70 or above à Reliable measure 8
8/15/17 Methods of Testing = Is a measure consistent? Multiple raters (i.e., researchers) should come up with interpretations that are highly correlated Good to use if a measure must be interpreted E.g., coded observations Validity Does a test actually measure what you think it does? E.g., Is our measure actually assessing intelligence or something else? Validity Validity: How accurately do the measurement items assess the latent? Validity: Does a measure, on the face of it, seem to measure what you think it should? Weakest form of validity and can be misleading Something can appear to measure a latent, but do it inaccurately 9
Validity Example: Item assessing prejudice, On a scale from 1 5, how much do you dislike the homeless? This item appears, on face-value, to be measuring prejudice But, what else might it be measuring? E.g., social desirability, lack of empathy, meritocracy beliefs Often, touchy subjects need to be measured Actual prejudice item, The government and media show more attention to issues of homelessness than this issue deserves Validity Validity: Does a measure assess all aspects of the? Prejudice E.g., A measure of prejudice needs to assess all aspects of prejudice Negative affect toward outgroup, stereotyping, discrimination This plan is termed the latent model Should encompass all aspects of the latent Negative Affect Discrimination Stereotyping Validity Validity: Does the measure predict behavior or other measures in expected ways? What should a good measure of prejudice predict? Negative non-verbal behavior (anxiety, anger) in intergroup interactions Stereotyping outgroup members Attempting to segregate from outgroup members Less willing to hire outgroup members as employees More willing to outgroup members Etc 10
Validity Types of Predictive Validity: Validity when a measure positively predicts another measure in an expected way E.g., prejudice measure positively predicts social distancing Validity when a measure negatively predicts another measure in an expected way E.g., prejudice measure negatively predicts number of outgroup friends Validity when a measure does not predict another measure that it wouldn t be expected to E.g., prejudice measure shouldn t predict extraversion Building a Nomonological Network Network of inter-related s If you have a valid measure, then the resulting scores should be related many other s in expected ways Number of Outgroup Friends Negative Non-verbal Behavior Divergent Validity Prejudice Measure Convergent Validity Discriminate Validity Willingness to Convict Pilot Testing Pilot Testing is used to assess the reliability and validity of surveys Can find out What minimum number of items yield good reliability? Good reliability: α =.70 or above Are measured items assess the latent? Do factors correlate with each other or other s as expected? Can also test different types of question wording, item-order, response sets, etc Which produces the highest reliability/ validity? 11
Survey Construction The goal is to produce a scale that Has high reliability Has high validity Is as short as possible (to avoid participant ) Can be administered to many different types of people with the same level of reliability and validity Building a Questionnaire Whenever possible: Use surveys It takes a lot of work to make a good survey, so don t invent the wheel twice! Sources for surveys Literature search E-mail authors for permission Ask faculty Check with library staff Search the internet (PsycINFO, ) 12