We now look at the remaining standards for specific study designs and methods, in particular the causal inference standards.

Transcription

1 We now look at the remaining standards for specific study designs and methods, in particular the causal inference standards. 1

2 After taking this module participants will be able to explain the terminology and concepts related to causal inference and summarize the principles of causal inference approaches to designing and analyzing observational studies. 2

3 As a reminder, the causal inference standards are one of three sets of standards, primarily relevant to observational studies. Causal inference methods is a set of conceptual and statistical approaches that has been developed over the last twenty- five years and has seen increasing use in medicine and in social science fields. Over the past ten years there s been a rapid development of these methods within the medical field and their use is thought to have improved validity of observational studies. 3

4 The basic idea behind the standards is that most observational studies are designed and analyzed in way that can identify associations or make predictions and correlations, but not identify causes. Yet, nearly all the time, we re interested in causes, not associations. 4

5 What is a cause? Its specific definition is that if B occurs in the presence of A and does not occur in the absence of A we say that A causes B. But we rarely have such a one-to- one correspondence. Usually, we re really talking about causal factors: a cause not the only cause. In terms of probability, if the probably of an outcome, given A, which may be a risk factor, is higher than the probability of the outcome, given B, another risk factor or the absence of the risk factor; then, all other things being equal, then Factor A is a cause of the disease or a treatment that s effective. 5

6 Let s take a first look at the standards for causal inference methods. And I would say upfront that many of them refer to complex statistical ideas or methods, such as covariate histories or propensity scores. B ut a certain pattern in these standards jump out and that will help us later understand what each standard is referring to. 6

7 The thing that jumps out is the use of terms like covariate and confounders, propensity scores having to do with comparability of the resulting groups or instrumental variables in the balance of covariates among groups. Throughout all of these standards there s a great concern about the comparability of the two groups that are being compared. 7

8 We saw this earlier, this slide from a previous discussion, talking about the flaws of observational studies and mentioning no control or comparison group or inadequate control groups or difference in baseline between the groups. And you remember that disturbing effects was an older term for confounders. In all of these examples the concern is about the allocation of patients to the groups that are being compared, the group that says- - that gets Treatment 1 or Treatment 2. 8

9 It s worth looking at what randomized trial does versus an observational study with respect to allocation. In an experiment, and randomized trial is an experiment, the investigator controls the individuals or material investigated, the nature of the treatments or manipulations under study, and the measurement procedures. By contrast, in an observational study, some of these features, and in particular the allocation of individuals to treatment groups, are outside the investigator s control. 9

10 When we have concerns about allocation we call the variables that messed things up confounders and the problems with the studies confounding by indication, inadequate control for confounders, selection bias, comparing apples and oranges, and so on. Confounding by indication as a term referring to the idea that if you just take patients who were given Treatment 1 and compare them to patients given Treatment 2, whatever the indications were for them to get Treatment 1 and Treatment 2 in the first place, could be confounders. A covariate is a variable that associated with the outcome variable or the dependent variable. S ome covariates are also confounders; that s why the term appears in the causal inference standards. 10

11 Whenever we think about allocation we should think about our concerns about studies that we read about and whether their results could be true. These are all real headlines from publications about observational studies: Study suggests TV watching lowers physical activity, alcohol is associated with a lower risk of arthritis, and magnets lesson foot pain of diabetics a study finds. In each case we can think of why the study might have been flawed in terms of allocation, that is, whether the two groups that were being observed were really comparable. Studies suggest TV watching lowers physical activity we wonder if it patients who watched more or less TV had lower physical activity or propensity to have lower physical activity in the first place. Similarlywith the other studies. 11

12 So this concern about allocation is a central concept underlying the standards. And what the standards are saying is to think carefully about the process of causal exposure, that is, the process of allocation. One way to do this is to keep in mind the randomized trial you d like to be able to do and use it as a mirror against everything- - every methodologic decision that you make in designing or analyzing an observational study. In the second approach, which is more mathematical, more statistical, is the counterfactual model. The counterfactual model permits the analyst to conceptualize observational studies as if they were experimental designs controlled by someone other than the researcher, quite often the subjects of the research. The counterfactual model is a framework in which to ask carefully constructed what- if questions that lay bare the limitations of observational data and the need to clearly articulate assumptions grounded in theory that is believable. 12

13 What is this counterfactual model? It refers to a fundamental problem in causal research and that is that for any particular patient we can only observe the outcome of one of the treatments. For example, if Patient 1 gets Treatment A and dies and Patient 2 gets Treatment B and lives, we d like to know what Patient 1 would ve done if he got Treatment B or what Patient 2 would ve done if she got Treatment A. B ut we cannot observe that. Those are the potential or counterfactual outcomes, the outcomes that we never had a chance to observe. Because it is impossible to observe an individual s response to both treatment and control, causal effects can t be observed directly. A researcher could only observe the actual outcome, the outcomes that can t be observed are called counterfactual. 13

14 So now if we extend this from individuals to groups we have a group of patients treated with Treatment A. They re the treatment group and a group treated with Treatment B, the control group. And we can observe the results of Treatment A in the treatment group, but we can t observe what would ve happened with Treatment B in that group. And similarly, in the control group, we can observe what happened with Treatment B, but what happened with Treatment A is counterfactual. We can t observe it. The best we can do it is compare the average effects of Treatment A to the average effect of Treatment B. This comparison is most likely to be valid if patients are randomized to treatments and if variables don t change much after randomization. 14

15 Many statistical methods and analytical approaches have been derived from this observation. We can look at a couple of these methods, although a comprehensive course and these methods could easily take a year of study. The methods that were talking about are to design and analyze observational studies in such way that the observational data emulate those from hypothetical randomized experiments with relatively well- defined interventions. To take just one example, when we have a large number of patients who have been treated one way or the other in actual practice and we want to pull some of those patients out to compare with how they did with different treatments we should use a new user design. That is, in order to avoid confounding and confusion that come from including people who have been on one or the other treatment for a long time we would identify in a database or a data set just those patients who are new to the two drugs or other interventions that we re studying. An older term related to this idea is the term inception cohort. The idea is that if you started at time t with a group of 100 people on Treatment A and at some time later, a couple of years later, you wanted to define a cohort of those people, you would only get those who have done well or had stayed on or hadn t died in the course of the two years before you start recruiting. So including people who are not new users means that you are missing people who have dropped out of a hypothetical cohort from some time in the past. That s the concept behind a new user design. There are many other examples of approaches that emulate a randomized experiment approach and apply that to an observational study approach. Causal inference methods also use an analytic effort to approximate the experimental paradigm that balances treatment and untreated or control groups on other factors. Let me put that a little differently: Instead of just talking about the baseline differences or similarities of the two groups, causal 15

16 inference methods use models, use statistical approaches or models, in order to get comparability of the two groups. An older approach to this is stratification. You identify a group of factors that might be potential confounders that might affect the outcome as well as affect the allocation to the two groups and many match people up from the two groups- - the larger groups of people who got each treatment. You match them up so that they re more similar than the entire group of people in both groups. More advanced and recent approaches are the use of propensity scores and instrumental variables. And we ll spend most of the remaining time explaining what those are. 15

17 A propensity score is the probability that a patient will receive a certain treatment, we estimated from observed variables. These variables can be patient characteristics. And we ask what variables affect the chance that a patient will be given a particular treatment instead of another one. There are several ways we can use propensity scores to make the treatment in control groups more comparable then they would be if we used all comers in actual practice. 16

18 So why would we want to use the propensity score? To correct biased allocation. There are reasons a patient gets one treatment instead of the other. Then maybe sicker or less sick or be in a subgroup that clinicians believe does better with one or another drug. The problem is that when we believe they may do better with one or another drug we can t be sure that it is because of the drug. It might be because the group that we think does well with a particular drug has a better prognosis in the first place. This type of confounding by indication is called channeling biased. Drugs with similar indications are prescribed for patients with different prognoses. In an early article about channeling bias the authors wrote, The claimed advantages of a new drug may channel it to patients with special pre-existing morbidity with the consequence that disease states maybe incorrectly attributed to use of the drug. Channeling bias is a common enough type of biased allocation that we ll use it as an example to see how propensity scores are calculated. 17

19 Suppose we went to look at the risk of patient making a suicide attempt with different antidepressants. The columns on the right show the different drugs we might examine and our question is How does that variable, the choice of drug, affect the outcome, suicide attempt or no suicide attempt? If it were that simple, if it were for one thing a randomized trial and we knew that the groups of patients taking these different drugs were comparable to begin with, we d be pretty much done here. We would just randomize to these different drugs and analyze the outcomes. But in an observational study we have to deal with the fact that there are other variables that it could affect the choice of drug and the outcome. 18

20 I just listed three examples here: gender, age and insurance status. It may be, for instance, that patients or clinicians prefer a certain drug in women or in younger versus older patients or that insurance might affect what drug patient is prescribed. These, of course, are called covariates and they re potential confounders. 19

21 In a propensity analysis in order to calculate propensity scores or estimate propensity scores we take these covariates and we ask the question, Do those variables influence what is prescribed? So even though before we called Drug X an Outcome Y, now we re doing an analysis of how these variables affect the choice of drug and we do the analysis in such a way that we can get a probability for each patient of the likelihood that they would get a particular drug. We ll use bupropion as the example of what we re trying to predict, bupropion use. 20

22 And so for each patient we get a probability for being prescribed bupropion. F or the first patient, who is a thirty- year- old male who is insured, the chance of getting bupropion was relatively low to begin with. For the second patient, a woman who is forty- five and uninsured, it was a very high probability of getting bupropion and everyone else is in between. S ome patients have similar propensities and, of course, I ve drawn at the example in a way that may not represent all combinations in real life. But look at Patients 1 and 6: They re both male, they re both about thirty, they re both insured, and they both got the same drug as it happens. Anyway their propensity for getting bupropion was identical, which makes sense. If the factors that are used to predict or to estimate propensity are gender, age, and insurance, they are very similar in those characteristics so they would have a similar propensity score. 21

23 Now if you were doing an analysis you might use a method called stratification to just pick out similar people or you might do matching, like to try to match cases to controls for patients taking one drug to patients taking another, and these two would be in the same group. If any patients with those characteristics- - male, thirty and insured- - had taken bupropion, you might use them as a control for these two patients. Propensity does the same sort of thing but it does it numerically and it has some advantages in some circumstances, which we will get to. 22

24 Now let s put this information on propensity back with what we had before, which were the treatments and the outcomes. Now we have a propensity score for each patient as well as knowing which treatment they actually got and the outcome. And the idea is that we would use the propensity score to try to tease out groups of people who got one drug and groups who got the other drug that were similar. And I ll show you now an actual study to see one way of how the propensity scores are used to do that. Before we go on, note that there s still our other covariates that we might want to adjust for and include. It doesn t- - using propensity doesn t mean that there aren t still covariates that might be included in an analysis, but the propensity takes the piece of it, a good piece of it, from several covariates and puts it all into one number that can be used in various ways to get similar groups. 23

25 So the example is also about suicide with different antidepressants. The question in this study is whether the three different classes of antidepressants- - SSRIs, tricyclics, and S NR Is- - have different risks of suicide. The population was a large population from British Columbia, Canada, who were new users of an antidepressant. The outcome was a combined suicide death or hospitalization rate and so it is a much lower rate for suicide attempts. In fact, you could consider it a rare outcome: four to nine cases per thousand people per year. 24

26 They calculated the propensities for the people taking each of those three groups of drugs and you can see that overlap. I m not saying which is which because these are not actual data. This just represents what they did. Then they took the place where the propensity scores for the three groups, the people who have gotten different drugs, overlapped. They trimmed the study sample just to those people who have propensity scores that were overlapped for all three drugs. This method of using propensity scores is called restriction. They restricted the sample to those that had similar propensity scores. They also used the method called high dimensional propensity. High dimensional propensity is a big data kind of approach that looks beyond those covariates or measured confounders that we think first about and looks for others that emerge from a huge data set and then includes them in the propensity equation. And so it is supposed to have a higher ability, than the usual propensity methods of identifying patients who are similar. Other ways to use propensity besides overlap restriction our propensity matching, propensity modeling, and weighting. Propensity matching- - kind of intuitive concept, you would take people who have similar propensity scores from one of the groups to another group- - to the other groups and you would match them. And so you would pull in people who perhaps weren t in this area of overlap of all three classes of drugs, but specifically that - a person who got each of the three drugs could be matched having the same propensity scores and you could build up a bigger sample that way. Modeling and weighting we re not going to talk about, but they re other ways of bringing the propensity score into the analysis of how X affects Y, how the drug choice affects suicide outcomes. This study, by the way, found that all three classes of drugs had a similar risk of suicide, even though at first glance there appeared to be differences, after doing the propensity analysis and picking comparable patients from the 25

27 three groups it turned out to have no difference in risk. One more thing before we leave this diagram: The propensity gives you kind of an idea of how patients with different drugs differ. If you were in a randomized trial, you would probably expect those three graphs to be identical. The fact that they re different verifies the idea that there might have been channeling bias. Now I say all that; keep in mind these graphs don t represent the data from that actual study. I m just illustrating the point with this schematic graph. 25

28 Now, let s turn to instrumental variables. They deal with allocation problems in a different way. An instrumental variable affects the treatment choice but does not affect the outcome, except via the treatment choice. Randomization balances confounders by allocating treatments to participants in a way that is unrelated to patient characteristics that are related to the outcome. Like randomization, good instrumental variable allocates treatments to participants and is unrelated to patient characteristics that are related to the outcome. So, it can distribute measured and unmeasured confounders more evenly across the treatment groups. In some circumstances, then, an instrumental variable could be used as substitute for randomization. Let s use some examples to illustrate the point. 26

29 Some instrumental variables that have been used for surgery or emergency interventions are distance to the hospital or the choice of healthcare provider. S o, in other words, some procedures just are not done that much among people who live further from the hospital. Or, in some cases hospitals have different practices and so where you live affects what treatment you get whatever your characteristics. So even though the patients aren t randomized the choice of treatment depends on how far they are from the hospital or what hospital they are closest to or go to, rather than on clinical characteristics that might lead the clinician to choose one treatment instead of another. Similarly, very closely related idea is that some emergency interventions can only be done within a short amount of time. We look at things like angioplasty or emergency stroke treatments, and so whether the patient gets that treatment or not depends on when they get admitted to a hospital in relation to the onset of their symptoms. And so, again, distance to the hospital or the choice of hospital may be affected by the timing of hospital admission and that may be a factor that influences allocation that isn t related to patient characteristics. Of course, it could be, but we can get into that later. 27

30 Here s some examples related to diabetes drugs. We re going to try to come up with at least one propensity factor. If we were doing a study of how different diabetes drugs affects outcomes and those outcomes could be cardiovascular outcomes, complications of diabetes or they could be things like markers of diabetes control, like hemoglobin A1c. Well, the obvious propensity factor is the one that we already discussed: marketing claims. When new diabetes drugs have come on the market over the years, they re usually accompanied by claims about who they are best for. F or example, use these particular diabetes drugs, not because they control glucose better, but because they have fewer metabolic side effects. So you should preferentially use them in people at high risk of cardiac disease. If clinicians believe the marketing claims and use them preferentially among- - in patients who have higher risk of cardiovascular disease it is obviously going to be hard to compare the patients who got the drug with those who got other choices in a study looking at cardiovascular outcomes. So marketing claims can lead to different propensities- - again, that s called channeling bias. But what are some possible instrumental variables? Well, some formularies may have covered the new drug faster than others. Then whether or not a particular patient started the new drug might be related to which formularies it was on, rather than to characteristics of that patient that might be related to outcomes. S o, the formulary could be an instrument. Past prescribing behavior is a very interesting instrumental variable. So if you noticed that people within particular medical group are using a lot of different choice, let s say diabetes drugs. Some are using more of certain insulins or more oral drugs and others are using other oral drugs. One thing you could do is you could look to see if that s their history, if that provider has a history of using more of that kind of drug than another type of drug. That would suggest that the patients 28

31 characteristics are not driving the choice of drug so much as the prescribers habit and that can be used to advantage. If the patients characteristics are not driving the choice of drug, while that s not randomization the groups of patients are likely to be more similar than if the prescribers were taking each patient and tailoring the therapy to them in a way that was the same across providers. So taking advantage of these quasi- experiments, is what instrumental variable analysis does. 28

32 Now let me make some summary points on propensity and instrumental variables. Propensity scores have this advantage of condensing numerous variables into one. They re also called a variable reduction method. They re good for measured confounders, but they re not good for unmeasured confounders. And randomization you have to remember, randomization has the advantage of balancing unobservable confounders. When we re looking at a randomized trial and trying to decide if it was done well- - if randomization was successful- - we look at the observable characteristics of the two groups. We see that they re similar in age, gender- - perhaps if it is a cardiac trial, their past history, their EKG findings at baseline and so on. B ut we look at those observed ones to try to make an inference of whether randomization worked, because it is really the unobserved ones that we re worried about and propensity scores can t do much about the unobserved confounders. And that can lead to problems. So, for example, suppose patients who get the newer drug for whatever reason- - let s say that we think that we channel them to the new drug for some characteristic, but suppose that they re also less amenable to lifestyle changes for this cardiovascular outcomes study and they have worse outcomes. Well, the poor outcomes probably aren t due to the drug. It is due to the fact that there was an unobserved confounder that at baseline we didn t have a baseline of how amenable they were to lifestyle changes. B ecause we didn t have that measure they turned out to do worse. So unobserved confounders are the Achilles heel of propensity scores. That s why methods that have developed to use more propensity scores more aggressively have put in the kitchen sink. They put in lots and lots of variables hoping to minimize the chance that there s unobserved confounders that haven t in some way been taken into account. In theory, though, instrumental variables can balance unobserved confounders. S o, in theory, 29

33 instrumental variable analysis is very powerful approach. In both cases though we have to check whether using the propensity or instrumental variable method actually balanced the known confounders, the measured covariates. So we need to look at whether all of this we did, looking at, let s say, using past prescribing behavior as an instrumental variable, did the groups that we composed using that approach look similar, have similar age, gender distribution and in all those other variables? Just like we would do for randomized trial: How well did it balance covariates? Does that give us any insight into the chance that it balanced unobserved covariates as well? And I should say this is adapted by a slide from Steven Pizer., The previous example using past prescribing behavior is described in great detail in the talk that this slide is adapted from. 29

34 So now let s look back at the standards for causal inference methods and see if we get a better idea of what they are about. CI- 2: Describe the population that gave rise to the effect estimate. This is the clinical thinking part of using these methods. As I said earlier, one of the key principles is to think carefully about how patients got assigned to different drugs in actual practice. Think carefully about those populations and what they re characteristics are and what the pitfalls and strengths might be of using a propensity instrumental variable, stratifying or other matching approach. CI- 5 and CI- 6 were for specifically propensity and instrumental variable, respectively. CI- 5 is to report the assumptions underlying the construction of propensity scores and the comparability of the resulting groups in terms of the balance of covariates and overlap. The balance of covariates we just talked about. We should look at the resulting groups and see if they re comparable on things like age, sex and other possible confounders. The overlap we also looked at in that graphic. Think of a situation where the overlap is tiny compared to the areas of non- overlap. So that the people who got one drug and the people who got another drug had very different characteristics to begin with. If you see a pattern it seems unlikely that it is going to be fruitful to look at the small sliver of patients who could ve gotten either drug, even though they may be comparable, they re so unrepresentative of the entire group and because the groups are so different, there may be a higher risk that unmeasured confounders make even those groups in the overlap area different. S o that s why the standard is there. With respect to the instrumental variable, the standard says to assess the validity of the instrumental variable that is how the assumptions are met, the assumption about it being independent of patient characteristics and independent of the outcome, and report the balance of covariates as well, which we also talked about 30

35 balancing covariates, looking againat those characteristics of the constructed groups. So these standards kind of come together once you know, at least, the background on propensity and instrumental variable standards. And I think the remaining standards, CI- 3 and CI- 4, in particular, which both refer to refining the timing of exposure, and measuring confounders before the start of exposure also make sense. CI- 1, defining analysis population using covariate histories, is not something we addressed. It requires deeper study to get to that as I ll show in the next slide. 30

36 So these are areas we didn t talk about today that are good for deeper study, if you want to get a deeper causal inference method. And the first one are causal diagrams, which are diagrams that have a mathematical meaning and can be very useful for understanding the mechanisms by which people got allocated to different groups and how to handle them. And it is a high- level area of statistics, but one that if you re working with a statistician, if they re familiar with it and adept at it, can improve and strengthen observational studies. The second thing we didn t talk about are the assumptions underlying causal inference methods, in particular underlying the counterfactual model that we referred to. And these also are mathematical assumptions that we re not going to discuss today, but which would be worth studying if you re very interested in learning more about causal inference and particularly about the potential outcomes our counterfactual outcomes model. And we also didn t talk about time- varying covariates or marginal structural models. These are mechanisms to deal with covariates that change over time, even after the study begins, even in a randomized trial, for instance. It may be that some covariates change over time. People might continue smoking, for instance at different rates. S o their risk- - that risk factor gets bigger or smaller as the study goes on. There are causal inference models, such as marginal structure models intended to deal with this problem and the standard I referred to, CI- 1, about covariate histories refers to these methods. 31

37 B efore we close, I want to bring up another central concept, and that is, don t overlook chances to strengthen the study design. The approaches I am talking about require some creativity and open- mindedness, rather than advanced statistics. S o, when you design a study, you should think, What specific items of information are needed to support or falsify the hypothesis? In designing a study, an investigator might start with the idea, I want to show that proton pump inhibitors [PPIs] are associated with pneumonia. And so the investigator designs a study to measure rates of pneumonia among patients who did or did not take a proton pump inhibitor. The investigator should go further, asking, What analysis would show that my hypothesis is false? The latter is more powerful approach to answering scientific questions. F or example, you could use the method of falsification hypotheses. In this approach, the investigator would not only ask, Is PPI use associated with pneumonia?, but also, Is the intervention associated with implausible outcomes? This is what Jena and colleagues tested in a paper in the Journal of General Internal Medicine. They studied whether using proton pump inhibitors was associated with community- acquired pneumonia and, in their analysis, they found that it was. Then they tested whether proton pump inhibitor use was associated with other outcomes that the investigators believed made no sense osteoarthritis, chest pain, urinary tract infection, deep vein thrombosis, skin infection, and rheumatoid arthritis and, again, it was with all of them. So, their conclusion is that there must be some kind of undetected confounding in the association between PPI use and pneumonia, even though we don t know what it is. The use of the falsification hypothesis suggests that it exists. There is an interesting editorial about this method in 2013 in JAMA by Vinay Prasad, MD, MPH, my colleague at Oregon Health & Science University. Underlying the PPI study are the concepts of biological 32

38 mechanism and specificity. Neutralizing stomach acid with a proton pump inhibitor might affect pneumonia. People who take a proton pump inhibitor often have reflux in the first place. And, if the reflux material goes into the lungs, perhaps it is more likely to cause an infection because neutralizing the acid in it permits more bacteria to survive. But there is no rationale for why PPI use would affect urinary tract infections and the other conditions that the investigators studied. If the association were specific to PPIs, the study would be strong evidence; but, the association was not specific to PPIs. A related strategy is to think about mechanism and temporality. If swelling, pain, and tenderness occur within minutes at the site of an IV delivering chemotherapy drug, we are very likely to say that the extravasation of the drug caused the reaction. The timing and our knowledge of the corrosive effect of that particular drug on tissues add to our confidence, even though I have not said anything about randomization, confounding, or even a control group. We can use mechanism and temporality to strengthen other studies, too. Joe Selby, MD, MPH, the [Executive] Director of PCORI, did this in a 1992 case- control study of sigmoidoscopy and mortality from colon cancer. The data source was medical chart review. The authors wrote, Because screening is not randomly assigned in a case- control study, selection factors related to both the likelihood of undergoing a screening sigmoidoscopic exam and the risk of dying from colorectal cancer may confound estimates of efficacy. Here is what they did about it: they classified fatal colon cancers according to whether they were within or beyond the reach of the sigmoidoscope. In their analysis, screening sigmoidoscopy was associated with a lower risk of death in the part of the colon and rectum that can be seen with the rigid sigmoidoscope, but there was no effect beyond the reach of a sigmoidoscope. In other words, they had a falsificationhypothesis based on the mechanism, namely the reach of a sigmoidoscope. There should be an effect in the sigmoid colon but not in the right and proximal transverse colon. As they concluded, the apparent benefit of screening by sigmoidoscopy was confined exclusively to the part of the colon and rectum that can be seen with a rigid sigmoidoscope. It is difficult to conceive of how such anatomical specificity could be explained by confounding. 32

39 Finally, I wanted to say, you can practice these approaches in actual clinical practice. Envision the studies that could be done, but aren t and potentially similar studies that have differences with respect to allocation, with respect to randomization, or how you would construct the groups. If you are doing research, ask the greatest skeptic you know to review the protocol. Don t rely on people and yourself who may not see a weak point in the analysis or may not think of the confounders that would underlie the credibility of the study. And for practice you can apply these ideas in assessing results in individual patients. Think of what covariates should be measured or could be measured when you give a patient a treatment and they respond, if you really wanted to disprove or prove that it was the treatment and not some other factor that affected the outcome. Think about your beliefs about tailoring therapy: Why you would use one treatment instead of another, one approach instead of another? And think about where those beliefs come from. Do they come from strong clinical evidence and randomized trials? Do they come from marketing claims? Think about where they come from and even how to test those beliefs. Because if those beliefs are tested, for example, if you believe one diabetes drug is better than another for patient with heart risk factors because it affects cholesterol, think of how to incorporate that into study design to see if the affect on cholesterol really was correlated with the effect on coronary outcomes. And think of instruments in the sense of instrumental variables that might make comparisons that are more valid and maybe not as valid as a randomized trial, but more valid than it would otherwise be in actual practice. For example, are there colleagues who have different beliefs or other factors affecting treatment allocation that aren t under your control or the patients control? S o, for instance, you might have two clinicians who have different philosophies- - let s say, 33

40 regarding cataract surgery, two ophthalmologists- - one who prefers to operate at the earliest sign of an opacity and the other who prefers to wait to see what the rate of progress is. The outcomes of the patients of those two clinicians may not have to do- - may not differ based on the characteristics of the patients. it is not a randomized trial, but it is a quasi- randomized trial in a sense of two different clinical philosophies: early operation versus late. Think of opportunities, natural experiments, that could be useful in designing observational studies. Thank you. 33