Experimental Methods in the Social Sciences Introduction to Experiments August 5, 2013
Experiments Increasingly Important in the Social Sciences Field experiments in political science as early as 1920s Growing use of experiments in development economics, transportation, education, criminology, Global experiments in economic development, alternative energy sources, education 2
Increase in Articles Using Experiments in American Political Science Review 3
Why More Experiments Now? Better technology Maturation of social sciences Increased understanding of the complexity and interdependence of research in social sciences Social sciences catching up to physical sciences and medicine 4
Key Elements of Experiments Standardization Randomization Treatment versus Control Groups Between-Subjects versus Within-Subjects Design Internal versus External Validity Experimental Bias 5
Standardization The Dependent Variable (DV) the thing to be explained is measured the same way in the same context across subjects Independent Variables (IVs) the explanatory variables also controlled and standardized Compared to field studies, experiments eliminate or control unmeasured influences on the DV and IVs 6
Treatment versus Control Treatment is the Independent Variable that the experimenter manipulates. The Treatment Group receives the treatment. The Control Group that does not receive the treatment An experiment may contain multiple treatments and a control group. 7
Blinded Experiment In a blind experiment, subjects do not know whether they received the treatment or control. A placebo is a false treatment, meant to make the control group believe they received the treatment. In a double-blind experiment, the experiment administrators do not know who received the treatment or control 8
Randomization The key feature of an experiment is randomization In laboratory and survey experiments, subjects assigned randomly to treatment and control groups If subjects volunteer for either treatment or control, we cannot be sure that different outcomes are due to selection bias or to the treatment 9
Randomization A simple randomizing device such as a coin flip can assign subjects to treatment or control. An n-factor design includes n simultaneous treatments. Each subject randomized across the n treatments. 10
Factorial Design Factorial design indicates the number of different treatments and conditions per treatment Each subject receives a combination of treatments (or controls) For example, a researcher may want to test whether a combination of: vitamins (vitamin or no vitamin) exercise (exercise or no exercise drugs (Drug A, Drug B, no drug) affect a subject s health 11
A 2x2x3 factorial design indicates three separate treatments. The first and second have two conditions (such as a treatment and control) The third has three conditions (such as a control and two treatments) The 2x2x3 design breaks subjects into 12 different groups for analysis. 12
Between-Subjects versus Within Subjects Between Subjects breaks subjects into treatment and control groups to measure differences between the two groups after treatment. Within Subjects design measures subjects before treatment and after treatment. Each subject is its own control. Within Subjects also a Pre-post design. 13
Randomized Controlled Trial (RCT) The most powerful experimental technique in scientific research RCT s are usually necessary in a clinical trial for medical treatments Randomized Trial similar to RCT but does not contain a control group 14
RCT s Classified by Study Design 1. Parallel Group: Each subject randomly assigned to a group. All subjects in a group receive the same treatment (or control) 2. Crossover: Each subject receives or does not receive the treatment in a random sequence 15
3. Cluster: Pre-existing groups of subjects are selected to receive the treatment (or control). Groups may include schools, classes, villages, city blocks, provinces, countries. 4. Factorial: Each subject randomly assigned to a group that receives combinations of treatments (or controls). 16
Standard Reporting Flowchart for Parallel RCT 17
18
Example: Dartmouth SPORT Study (Weinstein, et al.) Spine Patients Outcome Research Trial (SPORT) RCT to assess whether surgery or conservative care (rest, physical therapy, drugs) provided better outcomes for herniated disc, spondyliosis, and other ailments of the spine Published in the Journal of the American Medical Association, 2007, 2008 19
Surgical vs. Nonoperative Treatment for Lumbar Disc Herniation Weinstein et al 2791 2,720 Patients were screened 729 Patients were ineligible 426 Were not surgical candidates 19 Had fracture, infection, or deformity 129 Had inadequate non-operative treatment 20 Had cancer 135 Had other reasons 1,991 Patients were eligible 747 Patients declined to participate 501 enrolled in the randomized cohort 743 enrolled in the observational cohort 245 were assigned to surgery 256 were assigned to non-surgical treatment 521 chose surgery 222 chose non-surgical treatment 203 Were available at 6 wk 40 Missed the follow-up visit 2 Withdrew 0 Died 219 Were available at 6 wk 37 Missed the follow-up visit 0 Withdrew 0 Died 464 Were available at 6 wk 57 Missed the follow-up visit 0 Withdrew 0 Died 197 Were available at 6 wk 24 Missed the follow-up visit 1 Withdrew 0 Died 75 (31%) Had undergone surgery 46 (18%) Had undergone surgery 471 (90%) Had undergone surgery 4 (2%) Had undergone surgery 198 Were available at 3 mo 45 Missed the follow-up visit 2 Withdrew 0 Died 211 Were available at 3 mo 44 Missed the follow-up visit 1 Withdrew 0 Died 434 Were available at 3 mo 84 Missed the follow-up visit 2 Withdrew 1 Died 187 Were available at 3 mo 34 Missed the follow-up visit 1 Withdrew 0 Died 116 (47%) Had undergone surgery 73 (29%) Had undergone surgery 489 (94%) Had undergone surgery 19 (9%) Had undergone surgery 200 Were available at 6 mo 37 Missed the follow-up visit 8 Withdrew 0 Died 210 Were available at 6 mo 41 Missed the follow-up visit 5 Withdrew 0 Died 443 Were available at 6 mo 70 Missed the follow-up visit 7 Withdrew 1 Died 187 Were available at 6 mo 33 Missed the follow-up visit 2 Withdrew 0 Died 133 (54%) Had undergone surgery 96 (38%) Had undergone surgery 492 (94%) Had undergone surgery 35 (16%) Had undergone surgery 202 Were available at 1 yr 29 Missed the follow-up visit 14 Withdrew 0 Died 213 Were available at 1 yr 27 Missed the follow-up visit 15 Withdrew 1 Died 448 Were available at 1 yr 56 Missed the follow-up visit 16 Withdrew 1 Died 189 Were available at 1 yr 28 Missed the follow-up visit 5 Withdrew 0 Died 139 (57%) Had undergone surgery 106 (41%) Had undergone surgery 493 (95%) Had undergone surgery 44 (20%) Had undergone surgery 187 Were available at 2 yr 35 Missed the follow-up visit 23 Withdrew 0 Died 191 Were available at 2 yr 36 Missed the follow-up visit 27 Withdrew 2 Died 429 Were available at 2 yr 48 Missed the follow-up visit 43 Withdrew 1 Died 192 Were available at 2 yr 14 Missed the follow-up visit 15 Withdrew 1 Died 141 (58%) Had undergone surgery 110 (43%) Had undergone surgery 493 (95%) Had undergone surgery 48 (22%) Had undergone surgery 180 Were available at 3 yr 35 Missed the follow-up visit 29 Withdrew 1 Died 170 Were available at 3 yr 47 Missed the follow-up visit 37 Withdrew 2 Died 382 Were available at 3 yr 76 Missed the follow-up visit 60 Withdrew 3 Died 175 Were available at 3 yr 24 Missed the follow-up visit 22 Withdrew 1 Died 142 (58%) Had undergone surgery 111 (43%) Had undergone surgery 493 (95%) Had undergone surgery 52 (23%) Had undergone surgery 149 Were available at 4 yr 47 Missed the follow-up visit 33 Withdrew 1 Died 15 Had visits pending 144 (59%) Had undergone surgery 150 Were available at 4 yr 46 Missed the follow-up visit 43 Withdrew 2 Died 15 Had visits pending 115 (45%) Had undergone surgery 342 Were available at 4 yr 100 Missed the follow-up visit 76 Withdrew 3 Died 493 (95%) Had undergone surgery 166 Were available at 4 yr 28 Missed the follow-up visit 27 Withdrew 1 Died 53 (24%) Had undergone surgery Figure 1. Exclusion, enrollment, randomization, and follow-up of trial participants. The values for surgery, withdrawal, and death are cumulative over 4 years. For example, a total of 1 patient in the group assigned to surgery died during the 4-year follow-up period. [Data set 04/10/2008].
SPORT Study Has Many of the Same Problems Social Scientists Encounter Many subjects did not comply. Some assigned to surgery opted for conservative care. Some assigned to conservative care opted for surgery Many subjects could not re-contacted Potential placebo effect from surgery Some outcomes measured using surveys: People were asked how they felt 21
Internal versus External Validity Internally validity asks, Did the experimental treatment make a difference in this specific experiment. External validity asks, To what populations, settings, treatment variables, and measurement variables can this effect be generalized? 22
Internal Validity Threatened by... History Intra-experiment events (intrasession history) Selection bias in subjects Maturation of subjects Performance effects (conditioning) Regression toward the mean Mortality or attrition of subjects 23
External Validity Threatened by... Uncontrolled interactions, omitted variables, spurious correlations Testing may make subjects more sensitive than rest of population to variables under investigation Unrepresentative samples (college students in particular) 24
External Validity Threatened by... Hawthorne effects: subjects change their behavior when observed Professionalized subjects (Panel conditioning: experiment may change subjects) Mortality or attrition of subjects 25
Experimental Bias Expectancy Effects: the experimenter influences subjects Demand Effects: Subjects anticipate the purpose of the experiment Experimenter Bias. in recruiting subjects, selecting experiment time and place, coding and analyzing data. 26
Ethical Issues No physical, financial, emotional harm Subjects must give informed consent Subjects should not reveal information that is illegal or threatening to them Deceiving subjects acceptable as long as they are debriefed 27
Common Types of Experiments Laboratory experiments Field experiments Survey experiments Natural experiments Quasi-experiments: usually non-random 28
Other Thoughts You can solve only one hard problem at a time. Think about causation and mechanisms Start with a theory. Experiments test theories. Proof is theoretical. Confirmation is empirical. Read across academic disciplines To paraphrase Marvin Minsky: for any problem (research question), the best solution (experiment) is the one that uses the least time, energy, resources 29