Rival Plausible Explanations

Transcription

1 Ci Criminology i 860 Quasi-Experimentation and Evaluation Research Remember John Stuart Mill s 3 Criteria for inferring the existence of a causal relation: 1. Temporal precedence 2. Existence of a relationship 3. Elimination of Rival Plausible Explanations 1

2 Remember also how they were addressed in the classic experiment: Temporal precedence: Because you cause the cause to occur, and exert manipulative control over the setting, you can assess when the change happens. Relationship: You can assess whether, overall, the degree of change is beyond what you d expect on the basis of chance variation alone. Rival Plausible Explanations: These are handled by random assignment and control groups Can This Logic Be Applied Beyond the Experimental Context? Donald T. Campbell answered yes. He urged researchers to differentiate between the trappings of experimentation, and its inferential essence The name of the game was to move from manipulative control to analytic control 2

3 Quasi-Experimentation In quasi-experimentation you address the criteria slightly differently: First, you ask whether there really was a manipulation of the independent (treatment) variable. In not, then nothing to evaluate. Second you ask whether h there was a change in the dependent variable before and after the treatment variable appears. If there is no change, then there s nothing to explain. Quasi-Experimentation But if the answers to the first to questions are yes then the next thing to do is to ask why?. Then your job -- the application of analytic control -- is to think up all the rival explanations you can think of, and look for the relevant data that would allow you to evaluate their plausibility. 3

4 The Connecticut Speed Crackdown (Campbell & Ross) The first question is whether the treatment variable really was manipulated Figure 12.1 shows that the number of suspensions went way up. The Connecticut Speed Crackdown (Campbell & Ross) Maybe we should be a little cautious because of the parallel increase in not guilty resolutions, but, given the huge number of tickets, it s still pretty clear there was a big crackdown on speeding 4

5 The Connecticut Speed Crackdown (Campbell & Ross) Was there an effect? Well certainly the percentage of speeding violations went way down The Connecticut Speed Crackdown (Campbell & Ross) Was there an effect? And the decrease in the number of deaths on the roads was certainly in keeping with what you d expect. 5

6 Addressing the Third Criterion OK so far we re impressed. There really was a crackdown, and there really was a decrease in the number of deaths. But how do we know it was the crackdown that led to the decrease in deaths? The name of the game? Think up every rival plausible explanation you can think of, and get the data to test it out. Addressing the Third Criterion The inventory given to you regarding experimentation is useful here. For example, history is all events other then the treatment variable that could explain the change. For example Changes in weather? Changes in the roads budget? Changes in vehicle safety? 6

7 Assessing Regression Effects One threat that is often operative in evaluation research situations is regression toward the mean. With the Connecticut speed crackdown, we have to ask whether it might have occurred. Time series designs help Assessing Regression Effects Multiple time series designs can be even better because of they allow you to make comparisons with non-equivalent controls 7

8 Assessing Regression Effects Here is the same graph as shown on the previous page, but state by state. Notice that Connecticut is the only one that really does a major turnaround Addressing the Third Criterion Some new threats to internal validity that may arise in quasi-experimental settings: Diffusion (imitation) of treatment Compensatory equalization of treatment Compensatory rivalry Resentful demoralization Mortality 8

9 Can we go further than that? 9