OBSERVATION METHODS: EXPERIMENTS Sociological Research Methods Experiments Independent variable is manipulated, and the dependent variable respond to the manipulation. e.g. Eating a chocolate bar prior to a quiz increases quiz performance. Pretest Experiment Divide the group into two (experiment group vs. control group) Don t give any chocolate to the control group Experimental group gets the candy Retest- see if your manipulation (giving the chocolate bar) made an effect on the DV DV: quiz grades/performance IV: eating a chocolate bar Steps in the Classical Experiment 1. Formulate a hypothesis 2. Randomly assign participants to the intervention group or the control group 3. Measure the dependent variable(s) in one or both groups. This is called O1, or observation at time 1 4. Introduce the treatment or intervention 5. Measure the dependent variable(s) again. This is called O2 or observation at time 2. 1
Step 1 A research question that can be studied using the experimental approach A clear hypothesis about the relationship between IV(s) and DV(s) Based on confirmatory or explanatory rather than exploratory research questions Testing of new drugs- one IV one DV IV: taking vs. not taking a drug/ taking more or less of DV: getting better or not getting better/ the level of improvement in blood pressure Testing of a new fertilizer IV: some corn plants get a new fertilizer and some don t DV: Number of ears per corn stalk/ number of days it takes for the cobs to mature/ number of grams of carbohydrates Police training program IV: police who take part in a training program DV: Level of aggression in arrest Step 2 Need at least two groups Treatment group (intervention or stimulus group) Control group The treatment group (or groups) and the control group are involved in different experimental conditions In a true experiment, subjects are randomly assigned to the intervention or to the control group. To avoid systematic bias (to ensure that any differences between the groups are due to chance) Some people are more religious, some more wealthy, or less sickly, or more prejudiced- random assignment ensures that those characteristics are randomly distributed through the groups in an experiment Random assignment does not eliminate selection bias altogether, but makes differences between experimental groups due solely to chance Step 3 One or both groups are measured on one or more dependent variables. PRETEST DVs in humans, weight, height, attitudes, psychological states, mental and physical achievements Weight-loss programs: ratio of body fat to body mass Trying to raise women s understanding of the benefits of breastfeeding by having them watch a video presentation: women s attitudes about breast-feeding before they see the presentation 2
Step 4 The intervention (the change in the independent variable) is introduced. You manipulate the conditions. Step 5 The DVs are measured again. POSTTEST Example 100 college women (18-22 yrs of age) Measuring whether hearing the most popular rock song of the week improves/worsens performance on a task (how many 3-digit numbers each can remember) DV? IV? Intervention? Pretest? Posttest? 3
Criticisms Only women? No variation in category of student (no graduate students or high school students) No test of whether rock music helps or worsens learning more than other genres of music Doesn t test whether people can learn anything important or useful when they listen to or don t listen to music Confounds to Validity Suppose the rock listening group does better on the task Not because of gender, age or education, but because of the music. Introducing more independent variables without changing experiment design to control for these new variables creates CONFOUNDS TO VALIDITY. Confound (confuse) the experiment and make it impossible to tell what really caused any observed differences in the DV Critics of Experimental Research it forces you to learn more and more about less and less, until you know everything there is to know about nothing. However, narrowly defined questions, knowledge-making power Knowledge from a well-designed experiment; able to verify or falsify with another experiment. Example from before: repeated in another school? If you get a different answer, you need an explanation. Student selection process? Working class vs. upper-middle class people attending? Different socioeconomic classes grow up with different study habits or prefer different kinds of music. 4
The Classic Experiment Design IV(s) and DV Pretest (manipulation) Posttest Experimental groups vs. Control groups Clear operationalization Of both the IV and DV DV measured Stimulus presented DV re-measured 2 groups of comparable subjects Studies the effects of an IV on DV. Operational definitions are essential DV is measured, a stimulus is presented, and DV is re-measured. Changes in the DV are attributed to the stimulus There are 2 groups of comparable subjects. Only one group is exposed to the stimulus. Differences in the changes in the DV are attributed to the stimulus. Example Topic: effects of feeling-related vocabulary on peer interaction of preschool children. Hypothesis: A high level of knowledge of feeling-related vocabulary will result in an increase in feeling recognition and will therefore improve the peer interactions of 3-4 year old children. Experiment group DV: Peer interactions (pretest) Feeling vocabulary training DV: Peer interactions (posttest) Control group DV: Peer interactions (pretest) DV: Peer interactions (posttest) Other Examples of Classic Experiments Drug effects: one group receives a drug, the other receives a placebo Training effects Differential exposure effects Differential opportunity in experiments 5
The Solomon Four-Group Design Classic design prone to testing bias: differences in DV at Time 1 and Time 2 may be due to intervention, but may also be a result of people getting savvy about being watched and measured. Group 1: pretest-stimulus-posttest Group 2: pretest-posttest Group 3: stimulus-posttest Group 4: posttest Allows us to see if our pretest itself is causing a change in the subjects Solomon 4-example Sugar intake in children and acting out (behavior problems) Experiment group pretest sugar intake posttest (acting out) Control group pretest posttest (acting out) Experiment group sugar intake posttest (acting out) Control group posttest Natural Experiments Effects of large scale social/natural events on behaviors Media events Disasters Political events Problems Control groups may not be available Control groups may not be comparable Attrition may be a consequence of the stimulus and is selective Pre/post-measures may not be available e.g. earthquake depressive symptoms Experiment group and control group can t be comparable Can t have a pretest. 6
What can go wrong in experiments? Even though a causal research method, list of things can go wrong and we can t answer whether or not IV causes DV. Experimental effects may be Due to something else Internal validity problems Experimental effects may not be generalizable External validity problems Internal and External Validity True experiments using random assignment: high internal validity Changes in the DV probably caused by (not just related/correlated by) the treatment. With replication and verification: high external validity Generalize knowledge to people who were not part of the experiment. We are after replicated knowledge. Replication is as important as the production of knowledge. Kinds of Confounds History Maturation Testing and Instrumentation Regression to the mean Selection of participants Mortality/Attrition Diffusion of Treatments Demoralization 7
History Any IV other than the treatment 1. Occurs between pretest and posttest in any experiment 2. Affects the experimental groups differently During a lab experiment, the lights go out. If it goes out for both of the groups; no problem If it goes out only for one group; difficult to tell whether it was the treatment or the power failure that causes the changes in the DV. In a lab environment, history controlled by isolating the subjects; in experiments outside the lab, impossible to keep new IVs creeping in. Maturation Exposure to time may contribute to the outcome Anything that will happen because of passage of time Refers to the fact that that people in an experiment will grow older, get more experienced while the researcher is trying to conduct the experiment. Ex: start with a group of teenagers from a rural town and follow them for the next 60 years. Move to new cities Some to small towns Testing them periodically on DVs (political opinions, wealth, health, family size) But the people you are studying are GETTING OLDER. May become more conservative, wealthier, health may deteriorate Some of the changes in the DV is due to various treatments, some due to maturation Testing and instrumentation Testing: the process of pretest-posttest may influence the outcome The subject gets used to being tested for indicators on DVs, changing their responses Hawthorne effect: social and psychological phenomena tend to change due the experiment or the measurement itself (sleep studies) Placebo effect: anticipation of a change may bring about a change (medical experimentations) Instrumentation: results from changing the measurement instruments (changing the wording of questions in a survey is essentially changing instruments. Which responses do you trust?) Which observations do you trust? Yours or those of the new field researcher? (multi-researcher studies: interrater ability (training all investigators to see and record things in more or less the same way) 8
Regression to the mean Occurs when the study group has extreme scores on a DV No matter what the treatment is, you expect the extreme scores to become more moderate. If men taller than 2 meters marry women who are taller than 1,89 meters, their children will be Taller than average Closer to average height than either of their parents. (Can t really get more extreme than the height of the parents- hence, regressing to the mean) Happens on the aggregate, not on an individual level. Selection of participants Selection bias: in quasi-experiments and natural experiments In lab experiments, you assign subjects at random, from a single population, to both treatment and control groups Reduces the possibility that differences among the groups will cause differences in outcomes on the DVs Reduces, not eliminates the possibility altogether In natural experiments, we have no control over assignment of individuals to groups. Do migrants to cities from small towns engage in more entrepreneurial activities than stay-at-homes? If we could assign rural people randomly to the treatment group (migrating), then we would have some control, but we can t. Mortality/Attrition Refers to the fact that individuals may not complete their participation in an experiment 2 sets of married couples (200 couples in each set) for 5 years (family counseling as treatment) 1. 30 couples drop out, 130/170 (76%) still married 2. 50 couples drop out, 75/150 (50%) still married Lack of counseling caused the ones in the control group to get divorced at a faster rate? What about the ones that dropped out? Mostly still married? Mostly dropped. 9
Diffusion of Treatments Refers to when the control group cannot be prevented from receiving the treatment in an experiment Quasi-experiments where the IV is an information program e.g. diet and exercise behavior to lower BP. Demoralization Particularly for the control group Refers to when the control group knows that they are the control group; they know you are not going to do anything with them, so they lose interest. External validity problems 1. Sample problems: non-representative samples (you can t get your sample from Van because of logistics) 2. Control and experimental groups may not be comparable (Enka School vs. another one) 3. Experimental situation does not generalize to the real world Unrealistic lab circumstances Stimulus acts differently in the real world Interaction between pretest and stimulus 10