Question 2: Non-parametric Adjustment of an Observational Study

Transcription

1 Question 2: Non-parametric Adjustment of an Observational Study Jake Bowers and Paul Testa February 4, 2014 The saga of humanitarian treaties continues. The diplomat is very worried. She was meeting the Queen of England and ran into one of the knights, Sir David Cox. He read your report and said that he was compelled about your concerns about comparing like with like although he disagreed with the idea that simple overlap in a univariate sense is a strong argument that sides who sign treaties are being compared appropriately with sides in wars that do not sign treaties. He recommended that the diplomat read Rosenbaum (2010) (especially chapters 1, 3, 7, 8, 9, 13). That reading, combined with the reading from Gelman and Hill (2007) (especially chapters 9 and 10) made her feel like something more was needed it wasn t at all clear that the linear model was comparing like with like. It also wasn t clear which observations were causing the difference between the median and mean based approaches. She hired another analyst who produced the following results, again, only seen hazily on a torn piece of paper after the analyst defected to work for the newly rebuilt German version of the NSA after mumbling about balance and screaming about dfbetas. She knows that no adjustment observational study makes it the same as a randomized study. And, she wonders whether it is meaningful to consider the possibility that a given side might have shown different behavior (thinking about the treaty signing act) in a given war, and thus whether this relationship is properly understood to be telling us about a counterfactual causal comparison. Yet, a description of a relationship may still be useful, and she would really like to give a good recommendation to the president. Can you help her make sense of what the analyst left behind? It seems as if the analyst took two different approaches, each of which addresses a different problem that these data exhibit. The deep problem has to do with comparing like with like. But another serious problem involves the dramatic differences in description itself. Did this analyst improve on the linear model based adjustment from last time? Is there some way to combine the two approaches? If such a combination were made, would it be meaningful and provide substantively useful or policy-actionable information? Along the way: Make sure you know what Cook s distance is, how to calculate it without using a canned program, and how it might help simplify the problem of too many dfbetas. Remember that the diplomat has no use for rules of thumb. Propose perhaps some change to the linear model specification (recall the model the outlier with indicator variables or model the outlier techniques). Does that change the substantive interpretation? Make sure you can describe what is going on with the matching and balance checking. If you d prefer to try some of the other balance assessment ideas in Rosenbaum, feel free. Do you get different answers from them? Why would a person move from using the linear model to control for to this other kind of direct, non-parametric adjustment? What problems does it solve? What new questions does it raise? (Notice, for example, that we are comparing means of variables and we might prefer to do something else say, imagine using R s cut function or some other way to make this dependence on comparing means less restrictive.) How did this analyst represent the information about the matching in the linear model? What assumption(s) about the treatment effect is represented here (does the effect differ by matched set? should it? if you estimated an mean difference-by-set interaction what would you learn? how would you display what you learn?)? After you have figured out how to created a matched design using fullmatch and assessing balance using xbalance, you might want to inspect the matched sets directly to see if you agree with them. Maybe you d prefer a better configuration of like with like? Would a pair-match be better or worse? (See for example some discussion in Hansen (2004).) Make any changes that you think you ought to try. You might need a new, scalar distance matrix, or the exactmatch function, or the caliper function. References Gelman, A. and Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.

2 Question 2: Non-parametric Adjustment of an Observational Study February 4, Hansen, B. (2004). Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association, 99(467): Rosenbaum, P. R. (2010). Design of Observational Studies. Springer.

3 Question 2: Non-parametric Adjustment of an Observational Study February 4,

4 Question 2: Non-parametric Adjustment of an Observational Study February 4,

5 Question 2: Non-parametric Adjustment of an Observational Study February 4,