Confounds: Threats to Validity. Why Are They Important?

Transcription

1 Confounds: Threats to Validity Why Are They Important? Previously, we asked whether you would recommend that car magazine (Auto Tester Weakly) to a friend. We concluded that its recommendations were not valid. Validity Let's do a quick review of three common types of validity: Internal, External and Construct. 1. Internal Validity: When you think about internal validity, think INSIDE the experiment. Is your experiment so well designed that when the results are in, you feel confident that you can make truthful and definite statements about what happened in your study? If your study is relatively free of confounds; you will have high confidence in its results. That's internal validity. 2. External Validity: Can your results be generalized to people outside of your study? 3. Construct Validity: You are manipulating and measuring many concepts in your study are you really tapping into these concepts?

2 Most important to this discussion is Internal Validity and the concept of a confound. Most important to the concept of a confound is the concept of the independent variable. Independent Variable: The independent variable is the factor you manipulate. However, are you sure that the defined independent variable is the only factor influencing your dependent variable (the behavior that you measure or some other response measure)? If not, you have a confound. You have to think these issues through to make sure that you are truly (validly) manipulating the concept (or "construct"). No other factor should be operative. The Big Definition: Confounding occurs when two potential effective variables are allowed to covary simultaneously. Here's a big picture of the problem:

3 There are various types of confounds: Experimental Confounds: Nuisance variable whose levels are correlated with the levels of the independent variable. For example, you want to see if people of different ethnic backgrounds will be more likely to buy your car (they don't know it catches fire). You have them rate the attractiveness of the car. Did you control for income? Unfortunately, in the USA - discrimination has lead to different income levels. Thus, their car preference is confounded by income. Order effects in repeated conditions:

4 If you test everybody in the same order of the independent variable conditions - at the end of the experiment, you might just be measuring boredom, fatigue or the subjects get better because of practice over times. Differential Mortality: Because of some characteristic of the independent variable subjects, drop out. Say you want to study the effects of violent TV on mood for over a period of a week. You set up a protocol where your subjects have to watch certain programs. It turns out that women dislike watching violent TV. If you don't notice this, your results would be confounded by gender leading to differential mortality. Your results would only be valid for men. Demand Characteristics: Subjects figure out the point of the study and try to confirm or disconfirm you results - depending how they feel or merely being in a study influences them. Variants: Positive self-presentation: People want to be perceived as intelligent, moral, etc. and they may not respond in the typical fashion. Hawthorne effect: Experimenter effects: Solutions: Based on studies at the Hawthorne Works of the Western Electric Company. This effect is generally defined as the problem in field Ss' knowledge that they are in an experiment modifies their behavior from what it would have been without the knowledge. Expectancy Bias: Experimenters' hypotheses may influence outcome of experiment These are Classic in studies of ESP. Fundamentally, the experimenter cues the subject on how to respond. The cueing does not have to be conscious. Look up Clever Hans, the mathematical problem solving horse, or analyses of sign language using chimps or gorillas. The idea is that the experimenter may be unintentionally influenced by the experimenter's expectations.

5 Use double-blind procedures when designing and running the experiment. Bottom Line: Confounding is an incredibly easy concept. In a study, you manipulate an independent variable in order to measure its effect on a dependent variable. For your experiment to be valid, the only factor operating should be your independent variable. However, this can go awry and some other process or factor is influencing your study. When this occurs, your study is confounded. Confounding can occur when the experimenter accidentally manipulates the subjects in an unattended fashion, the subjects are influenced by merely being in an experiment and this is mistaken for a treatment effect, the groups are selected such that there is a bias between groups - and many other possible screwups. Appropriate procedures exist to test for and to control these effects such that your experiment will be valid.