Threats to validity in intervention studies Potential problems Issues to consider in planning
An important distinction Credited to Campbell & Stanley (1963) Threats to Internal validity Threats to External validity Problems we might encounter in this particular study. Problems that might limit the generalizability of the results Was it really the new program (the intervention) that caused a difference, or was it something else? Okay, the program seemed to work in this study, but would it work elsewhere? Next time? With new students, teachers, etc.?
Threats to Internal validity Outside events (History) Attrition Maturation Selection Practice (testing) Instrumentation Regression (statistical regression) Selection-Maturation Experimenter effects Subject expectations Diffusion
Outside events (History) Problems with issues that happen out in the real world Events might influence the outcome we are studying, but they are not really part of our new treatment News events The Sesame Street example There are almost always lots of factors that are influencing things, so other factors can have an influence on our participants Control groups can help deal with this
Attrition Drop outs A big issue in longitudinal studies College studies are a classic example Differential attrition Can make new programs look good e.g., if weaker students drop out Can make new programs look bad e.g., if weaker students are retained Or if weaker students drop out of control group Not always a problem in short-term
Maturation Normal development may occur while the new treatment is being studied An important issue with young children Comparison groups can help
Selection effects In comparative studies How are people selected for the treatment and nontreatment conditions? Volunteers? Intact groups? What biases might be present? This is the classic problem we encounter when we do not have a randomized experiment and have to use an observational study or a quasi-experimental design.
Practice effects Can occur with tests Also a potential problem with interviews Observations Other situations? Can be tricky to control
Instrumentation Changes in the instruments Different tests Different questions (in interviews or questionnaires) Different interviewers Different observers Different scoring systems Different raters
Regression to the mean Regression artifacts High scores tend to be not quite as high Low scores tend to be not quite so low Only an issue in special circumstances Can have important effects The Head Start example Here is a link to a nice discussion of regression to the mean: http://onlinestatbook.com/2/regression/regression_to ward_mean.html
Selection-Maturation interaction Even if two groups appear to be similar at the beginning of the study (at pretest), they may be growing at different rates. (Or their future growth may be different.) This can be a challenge for quasi-experimental designs.
Expectancy effects Experimenter effects Even unintentional influences can play a role One group may be influenced more than another Subject expectations (participant expectations) Hawthorne effects (just being in a study can be an influence) Knowing about new treatment may be an influence Compensatory rivalry in control group Demoralized control (knowing about control group may be an influence) Placebo effects in medical trials How to control these?
Diffusion of treatment Some of the new treatment ideas might diffuse (spread) to others Teachers may share ideas with other teachers So control group is not quite what you wanted People may find alternative ways to gain access to some new program or idea (This might be desirable in some ways, but it can certainly be a limitation for research studies.)
Treatment implementation Was the new treatment really implemented? (What was the Fidelity of the treatment implementation? ) Sometimes the control group implements some of the new ideas (treatment diffusion) Other times the treatment classrooms may not fully implement the new ideas A tricky issue that we run into in lots of studies especially studies that take place over long periods of time. Things can get better over time Or things might get worse when the novelty wears off
Threats to External validity Limited samples Special arrangements Expectancy effects Sensitization Multiple treatments