EXPERIMENTAL DESIGNS I: Between-Groups Designs There are many experimental designs. We begin this week with the most basic, where there is a single IV and where participants are divided into two or more groups to reflect the variation of the manipulated IV, and where we will compare the effects of the manipulation on the DV by comparing the different groups, while trying to hold everything else constant. This is known as a between-groups design, and it is the same as the differential design we covered three weeks ago, except that the IV is manipulated. I. Variance and the F Test The essential focus of an experiment is to show that variance can be produced in a DV by manipulation of an IV. The crucial step is to measure the DV, and if differences occur, this is known as systematic variance.
Systematic variance comes from two sources: experimental variance (the variance produced by the experimental manipulation of the IV) extraneous variance (any variance that can be due to the influence of extraneous variables) In addition to systematic variance, there is always a certain amount of variance in the DV that can be attributed to random factors that affect some research participants in an experiment but not others, and to all the various individual differences that can never be fully known or controlled; these are known as error variance. To support an experimental causal hypothesis, we need to be able to show that the systematic variance is greater than the error variance. Specifically, we calculate a statistic known as F: F = (SV+EV)/EV F will always be a number equal to or greater than 1.0 why?
The larger the value of F, the greater the likelihood that we have significant findings why? Experimental research is the highest constraint form of research; what makes it high constraint are the steps taken to: maximize experimental variance, control extraneous variance, and minimize error variance. II. Designing Good Experiments How do we maximize experimental variance? Have a solid basis in prior theory and research to support belief that IV should influence DV Make sure the IV really varies the two or more conditions should be substantially different If possible, conduct manipulation checks to be sure (note that a manipulation check does not involve the DV; it is designed only to ensure that there was a true difference between the conditions of the IV)
How do we control extraneous variance? Use as many of the control procedures discussed last week as possible Most important, use unbiased assignment of participants Sometimes, it is advisable to turn a confound into another IV, especially non-manipulated variables such as gender or age (when there is more than one IV, we have a factorial design,which we will examine in two weeks) How do we minimize error variance? Follow standardized procedures Use reliable measuring instruments Choose very homogenous samples but that limits external validity so it may not be a good choice III. Designs Designs that look like experiments but that are non-experimental: 1. Single-group designs (posttest only as well as pretest-posttest 2. Differential designs 3. Natural control group designs
All of these lack true experimental manipulation, and such designs do not adequately support causal inferences. True between-groups experiments require multiple groups (at least Experimental and Control) reflecting the manipulation of the IV, and where the experimenter controls the assignment of participants to groups through some form of unbiased assignment procedure. 1. Posttest only, two-level 2. Pretest-posttest, two-level (the pretest provides a check on the initial equivalence of the two groups, and also allows for the pre-post change to become the DV) 3. Multilevel (both posttest only and pretestposttest) 4. Solomon s four-group design to control for the possible confounding effect of pretesting
IV. Statistical Analyses Experiments usually involve score data (interval or ratio) for the DV, and we assess the effect of our experimental manipulation by looking for differences between groups in the DV. To determine whether we have a statistically significant difference, we use the t test when we have 2 groups, and the F test (ANOVA) when we have more than 2 groups. When we have more than 2 groups and use the F test, a significant F tells us the groups are different, but to determine exactly where the differences lie (e.g., A is different from B which is different from C, or A is different from B, but B is not different from C), we use t tests to conduct comparisons between any pair. V. Limits of Experimental Designs Experimental designs are the highest constraint, but as is always true, there are limitations. Because a good experiment is conducted under very controlled artificial conditions, often with very carefully selected participants, and with very rigorous and obtrusive measures of DVs, we can achieve very high levels of internal validity, but at the expense of external validity.
Thus, as we learned earlier, one of the purposes of field research is to test experimental findings outside the lab. And in addition, experiments will raise many more ethical concerns with respect to any risks that participants are exposed to through the manipulation. Finally, some IVs cannot be manipulated.