Measurement. Reliability vs. Validity. Reliability vs. Validity

Transcription

1 MEASUREMENT Outline 1. Measurement - Reliability vs. Validity - Validity - Threats to Internal Validity - Construct Validity (measurement validity) - Face validity - Content validity - Predictive validity - Convergent validity - Discriminant validity 2. Lab Methods Section of a Paper & Data set design and data entry Measurement Reliability vs. Validity If a study is reliable, it is repeatable. That is, no matter how many times or who does the study you will get the same results. If a study is valid, your study correctly reflected what is as close to the truth as you can get. There are different kinds of validity Reliability vs. Validity Reliable, but Not Valid Not Reliable, Not Valid Not Reliable, but Valid Reliable and Valid Real Study Example: Depression is the target, or the construct to be measured. Results of the different studies are the arrows. If it is Valid, it equally represents the target, depression. If it is reliable, each time depression is measured, the arrow lands in the same spot (consistency). 1

2 Types of Validity (see chapter 1) 1. Conclusion validity Are the study variables related? 2. Internal validity Is the relationship among the variables causal? The 3 factors 9 Threats to internal validity (problems reducing extraneous variables). 3. Construct validity Do our operational definitions of your variables and the measurement of your variables accurately reflect our construct (e.g., depression) 4. External validity Can we generalize our study findings to the population, other samples, places, time, etc. Threats to Internal Validity If your study has internal validity, it has established causation But, one factor of causation is elimination of external variables What are some threats to internal validity of a study what are some external variables? 9 Threats to Internal Validity 1. History 2. Maturation 3. Testing (practice effects, reactivity effects) 4. Instrumentation (instrumentation decay) 5. Statistical Regression 2

3 9 Threats to Internal Validity 6. Selection 7. Mortality 8. Interactions with Selection 9. Diffusion or Imitation of Treatments External Validity External Validity How well your study generalizes to the population. Termed external, because it concerns how the study represents the population (outside of the study) Can the results of your study hold true for other people, places, times, etc.? Good external validity depends on a good sample! How do we obtain a good sample? Random Selection of participants from a defined population. Construct Validity Construct Validity How well the operational definitions of your variables and the measurement of your variables accurately reflect our construct (e.g., depression). Let s provide some operational definitions of depression. More ways of measuring depression, the more likely we will be able to hit the target 3

4 Construct Validity Class Activity: Suppose you are a lifetime prisoner in a maximum security prison. Paper-and-Pencil Depression measures. 1. Face Validity on its face does it look like your measure reflects the construct, depression? Can you verify that a measurement of depression has face validity by giving it to a group of people diagnosed with depression, those recovering, and those who have never had depression. Does the measure distinguish among these groups? If yes, it likely has face validity. 2. Content Validity Create a sort of checklist of all the behaviors, cognitions, and emotions that would accurately represent depression. Check this list against your measure. Does it meet your criteria? Might also ask experts to define depression and use their checklist. OR Give you measure to experts and ask them if it meets their criteria for depression 4

5 The next 4 types of construct validity you check your measure against another measure to determine if your operational definition of depression is accurate. How the next 4 are different, is the criteria that are used to determine if your measure has good criterion construct validity. CESD > 16 (N= 20) CESD < 16 (N = 40) No Depr 10% N = 2 75% N = 30 Depr Dx 90% N = 18 25% N = Predictive Validity How well your measure predicts the actual occurrence of your construct (e.g. depression) Ex: Give the CESD to a group of newly admitted prisoners. Are those with initial higher scores, more likely to be diagnosed with depression later? 4. Concurrent Validity Does your measure distinguish between other groups of people who do not have depression, but perhaps another diagnosis with similar symptoms. Give the measure to people with depression, bipolar disorder, etc. 5

6 5. Convergent Validity That your measure of depression is positively related to other measures that it is expected to be related to. Ex: Scores on the CESD has a moderate positive correlation (.67) with convergent scales, such as pessimism, negative mood, etc. * Note: You don t want your correlation to be too strong (r =.80 or higher), or one could say that your measure essentially is no different than these other measures. Then why even use your measure? 6. Discriminant Validity How different is your measure from other measures that are not related to depression. Here, you look for no relation or a small to moderate negative relation (r = -.40). Ex: Depression and optimism. 6