Are Observational Studies Just as Effective as Randomized Clinical Trials?

Transcription

1 Blood Purif 2000;18: Are Observational Studies Just as Effective as Randomized Clinical Trials? Tom Greene Department of Biostatistics and Epidemiology/Wb4, The Cleveland Clinic Foundation, Cleveland, Ohio, USA Introduction The question of whether observational studies are just as effective as randomized clinical trials appears to presume a competition between the two. However, from a methodological perspective, the two types of studies are complementary. Observational studies and randomized clinical trials can be viewed as expressions in the setting of modern clinical research of the steps of observation and experimentation that form the basis of the scientific method. From this perspective, the observational step is used to uncover patterns and formulate hypotheses regarding cause-and-effect relationships. This is followed by the experimentation step in which the hypotheses formed in the observational setting are confirmed or refuted in an experiment in which the independent variables are controlled by the experimenter [1, 2]. Since both the observation and experimentation steps are required for scientific advancement, it seems misplaced to argue that one is more effective than the other. Perhaps the question of whether observational studies are as effective as randomized trials is intended to address whether the experimentation step constituted by randomized trials could be replaced by observational studies without impairing biomedical advancement. The central role of experimentation in most scientific advances over the last several centuries suggests that if the question is expressed in this way the answer must surely be no. As reviewed below, experimental investigation by randomized clinical trials provides a level of confidence in causeand-effect inferences that can rarely be approached by observational study alone [3, 4]. The full story is more complex than this, as in addition to observational studies and randomized clinical trials with human participants, biomedical evidence is drawn from other sources, including deductions from the existing body of biomedical knowledge and experiments based on tissue cultures and animal models. In some cases, the combination of experimental evidence from tissue cultures and animal models along with evidence from observational studies of humans has provided compelling cases for cause-and-effect relationships without the benefit of randomized trials. A classical example is the identification of cigarette smoking as a cause of lung cancer, an area where randomized trials in humans cannot be ethically performed. It is also important to recognize that the characteristics of observational studies vary widely, and that some limitations of cross-sectional observational studies can be overcome by longitudinal follow-up of a cohort over time. Moreover, due to stringent ethical requirements, financial limitations, and other feasibility issues, it will never be possible to address all therapeutic questions with randomized trials. This assures that in addition to their natural role in hypothesis generation, observational studies will continue to be relied upon for evaluation of therapies in situations where randomized trials are not ethical or are not feasible. Within this setting of complementary roles for randomized trials and observational studies, I will reiterate in the following sections the differences in the strength of causal inferences between these two types of research. ABC Fax karger@karger.ch S. Karger AG, Basel /00/ $17.50/0 Accessible online at: T. Greene Department of Biostatistics and Epidemiology/Wb4 The Cleveland Clinic Foundation, 9500 Euclid Ave. Cleveland, OH (USA) tgreene@bio.ri.ccf.org

2 Limitations in Causal Inferences from Observational Studies The central limitation of observational research derives the point that while the observational setting provides an excellent basis for characterizing associations between different variables and for formulating hypotheses, it is usually problematic to make cause-and-effect inferences the absence of experimental confirmation of these hypotheses. Many of the limitations in forming causal inferences from observational studies are due to three sources of bias: (1) uncontrolled confounding factors; (2) selection bias, and (3) reverse causality. For concreteness, suppose that we wish to determine the effect of a particular treatment on a specific patient outcome such as mortality. Each of the three types of bias relate to the point that without experimental confirmation, it is very difficult to determine whether an observed association between the treatment and the outcome is due to the hypothesized treatment effect or to an alternative explanation. Uncontrolled Confounding Factors The observed relationship between the treatment and outcome may give a biased estimate of the true treatment effect due to the joint association of both the treatment and the outcome with common extraneous factors, called confounders [5]. Such confounding may either lead to a spurious association between a treatment and outcome where no causal relationship exists, or it may obscure a true causal relationship by producing an association in the opposite direction. An example is provided by observational studies comparing the efficacy of continuous renal replacement therapy (CRRT) vs. intermittent hemodialysis in reducing mortality in acute renal failure patients. Since CRRT is often provided to patients with especially severe medical conditions, it is easy to see that patients treated with CRRT may have a higher mortality rate than those treated with intermittent dialysis, even if CRRT were the superior therapy. Similar issues affect other comparisons of alternative treatment modalities (e.g. peritioneal dialysis vs. hemodialysis) where the choice of treatment is likely to be related to the condition of the patient. The above example might be countered by the assertion that modern epidemiological and statistical methods may be used to remove the effects of bias due to known confounding factors. As described by Wolfe, these methods attempt to remove confounding by assessing the association of the treatment and outcome at fixed levels of the confounding factors. However, such statistical or epidemiologic adjustments require first that all important confounding factors be known by the researcher, and second that each of these factors is accurately measured and available for use in the statistical analysis. In the complex setting of the intensive care unit, it will be difficult to assure that any measured set of factors would account for all the differences between patients assigned to CRRT and intermittent dialysis that predict mortality. Selection Bias The relationship observed between the treatment and outcome may also give a biased assessment of the treatment effect if the mechanism for inclusion of patients is related to the outcome variable [6]. This issue is especially important in cross-sectional studies of prevalent dialysis patients, since subgroups with high mortality tend to be underrepresented because higher proportions of these patients will have died prior to data collection. An example of this phenomenon is illustrated in figure 1, which relates serum albumin to years of hemodialysis for a cross-sectional sample of 1,746 patients screened for entry into the HEMO Study [7, 8]. The mean albumin level increases as a function of the years of hemodialysis, both with and without statistical adjustment for demographic factors. The selection bias in this example is transparent; since patients with low albumin levels have substantially elevated mortality rates, only those with higher serum albumin are likely to have survived long enough to be included in the subgroups with longer durations of dialysis. Thus, it is likely that the positive relationship between serum albumin and years of hemodialysis is not due to a beneficial effect of long-term dialysis, but is rather an artifact of the selection bias. Reverse Causality Finally, the association between a treatment and outcome may give a biased estimate of the treatment effect if the outcome itself directly or indirectly affects the treatment. A transparent illustration of this problem is provided by any cross-sectional assessment of the association between the level of blood pressure and the use of hypertensive medications in a sample containing a significant proportion of hypertensive patients. Figure 2 shows the cross-sectional relationship between the level of mean arterial pressure (MAP) and the number of classes of antihypertensive medications for screenees for the Modification of Diet in Renal Disease Study [9]. As can be seen, without covariate adjustment MAP has a positive relationship with the number of classes of antihypertensives. This positive association is attenuated with adjustment Blood Purif 2000;18: Greene

3 Fig. 1. Cross-sectional association of serum albumin and years of dialysis among 1,746 participants at entry of the HEMO study. P = Unadjusted mean B SE serum albumin; v = adjusted mean B SE serum albumin levels controlling for age, sex, race, and renal diagnosis. Fig. 2. Cross-sectional association of MAP with number of antihypertensive medications among 1,555 patients at entry to the MDRD study. P = Unadjusted average mean arterial pressure (MAP); v = adjusted mean MAP levels controlling for age, sex, race, renal diagnosis, BMI, LDL cholesterol, HDL cholesterol, and smoking. for the covariates indicated in the figure legend, but the expected beneficial effect of the antihypertensives on MAP is still not apparent. The likely explanation is that the use of antihypertensives is determined by the blood pressure level in the first place (e.g. patients with higher blood pressure are prescribed more antihypertensive medications), and that this bias was only partly removed by statistical adjustment for the indicated covariates. More generally, the potential for bias from reverse causality limits inferences that can be drawn from any observational associations between a putative risk factor and a measure of disease outcome if it is possible that the disease itself may influence the risk factor. This problem is relevant to observational studies of chronic renal disease reporting associations between measurements of blood pressure and rates of progression, since it is recognized that renal impairment may itself adversely affect blood Observational Studies and Randomized Trials Blood Purif 2000;18:

4 pressure. An analogous issue confronts inferences from observational studies of dialysis patients showing strong associations of serum albumin and other biological markers of nutrition and/or inflammation with mortality and morbidity. In these studies it is likely that a decline in serum albumin is in part a response to the same disease processes which lead to an increased risk of mortality and hospitalization. Thus, it is not clear from the observational associations to what extent a therapy that succeeded in modifying the level of serum albumin would also reduce mortality and morbidity. The Power of Randomization In an ideal experiment, the independent variables are controlled by the experimenter to determine if these manipulations produce the hypothesized effect on the outcome variable within a highly controlled setting where variations due to extraneous factors are eliminated. In the clinical setting, such control of extraneous factors is usually not possible due to the wide variation among the human participants. The methodology of modern clinical trials overcomes this difficulty by randomly assigning a common group of patients to different predefined treatment protocols, and then comparing the outcome between the randomized groups [3, 10, 11]. In well-conducted clinical trials, randomization prevents systematic confounding and selection bias by assuring that the patients assigned to the respective treatment protocols are approximately equivalent with respect to all extraneous factors, regardless of whether these factors are known to the researcher. Further, randomized assignment assures that the selection of treatment protocol is not related to the outcome variable, preventing bias from reverse causality. While randomized assignment eliminates such systematic bias, it does leave open the possibility of imbalances between the treatment groups due to the randomization itself. However, the effect of random imbalances can be rigorously quantified by statistical concepts such as p values and confidence limits, and can be controlled by the use of a sufficiently large sample size. Difficulties in Quantification of Uncertainty in Observational Studies In the setting of observational studies, p values and confidence limits quantify uncertainty due to sampling error that arises from the use of a limited sample rather than the entire patient population. However, the p values and confidence limits to not account for the additional uncertainty due to the inherent susceptibility of observational associations to confounding from uncontrolled extraneous factors, selection bias, and reverse causality. Moreover, while sensitivity analyses for the effects of such biases are occasionally reported, in the large majority of cases it is difficult to quantify the risk of these biases. Thus, not only do results from observational studies have a greater susceptibility to bias from confounding, but the added uncertainty due to the risk of this bias is difficult to quantify. To be fair, probabilistic statements produced by randomized clinical trials are also limited in that they formally apply only under the conditions of the clinical trial itself. The uncertainty association with generalizing the results of a clinical trial to other settings (the problem of external generalizability) is not reflected in the usual p values and confidence limits, and must instead be assessed by comparing the circumstances of the trial with the clinical setting in which a therapy is to be applied [12, 13]. Strict Standards for Randomized Clinical Trials Allow Less Margin for Error Modern randomized clinical trials are conducted under narrow criteria which are often monitored by regulatory agencies, internal review groups, or specially formed external advisory committees [3, 10, 11]. In addition to strict criteria for assuring the safety of trial participants, standard criteria for randomized trials include: E The designation of a single, clinically relevant primary outcome variable (ideally a hard endpoint such as death which can be objectively ascertained). E Precise specification of the primary study hypothesis(es). E Precise specification of inclusion and exclusion criteria. E Precise criteria for stop points (e.g. termination of treatment protocols for individual patients due to side effects of the development of new medical conditions). E Blinded assertainment of outcomes. E Implementation of quality control procedures. E Monitoring of adherence to the treatment protocols. E Specification of the analysis plan prior to data collection. E Intent-to-treat primary analysis in which patients are analyzed according to their randomized assignment Blood Purif 2000;18: Greene

5 The standardization resulting from enforcement of these strict criteria facilitates the interpretation and comparability of trial results. While similar criteria could in principle be applied to observational studies, in practice looser standards are often applied in both the design and analysis phases. For example, in the observational setting the primary analysis is rarely specified prior to data collection. Often there are many choices in the data analysis as to how to model the effects of the treatment and adjust for potential confounding. Frequently, these choices have a substantial affect on the appearance of the results, leaving open the possibility that biases or erroneous judgements by the researchers may have colored their conclusions. The greater latitude in the conduct of observational studies can be advantageous from the standpoint of facilitating creative hypothesis generation, but can increase confusion when observational studies are used to make cause-and-effect inferences in the absence of randomized clinical trials. Illustrative Example: Observational Studies and Randomized Trials on Dose of Dialysis The interaction between randomized trials and observational studies can be illustrated by the collection of studies that have addressed the association of dialysis dose with outcome over the past 20 years. The National Coorporative Dialysis Study (NCDS) first established the importance of dialysis dose in influencing patient outcome by showing that patients randomized to dialysis prescriptions targeting a lower time-average concentration (TAC) of urea experienced fewer morbid events than patients randomized to prescriptions targeting a higher TAC [14]. Subsequent post-hoc analyses of the data from the NCDS suggested that dialysis dose was better quantified by the fractional removal of urea during dialysis, quantified as Kt/V [15]. These analyses led to a recommendation of a minimum Kt/V of 1.0. It is interesting that while the analyses of the NCDS in terms of Kt/V were made possible by the care taken in the context of a randomized trial to quantify and monitor the dialysis treatment interventions, these analyses were themselves observational since comparisons of different levels of Kt/V did not reflect the randomized treatments of the study. After the NCDS several observational studies reported an inverse association of Kt/V with mortality for Kt/V levels ranging to at least 1.2 [16 20]. Limitations in several of these reports pertaining to the precision of characterization of dialysis dose have been pointed out [7, 21]. From the standpoint of this debate, it is instructive to consider the biases noted above for cross-sectional studies relating Kt/V to outcome. These studies are subject to a risk of confounding since the status of patients receiving higher doses of dialysis may differ from those receiving lower doses. For example, a higher dialysis dose may be a reflection of a general pattern of superior medical care which includes components other than the dialysis prescription. Reverse causality is especially hard to rule out, since dialysis prescriptions are often increased in response to deterioration in a patient s condition. Even if a fixed dialysis prescription is used, progressive malnutrition may lead to a reduction in the volume of urea distribution V, which in turn leads to a higher Kt/V. These last two scenarios could cause a cross-sectional estimate of association to underestimate the true effect of dialysis dose. Conversely, under other plausible scenarios an adverse outcome may cause a reduction in Kt/V, resulting in a positive bias in which the effect of Kt/V is overestimated. For example, cardiovascular disease often causes conditions that interfere with the achievement of high blood flows, leading to lower dialysis dose. Some patients with deteriorating health also may reduce their compliance to their prescribed treatment times, and thereby reduce their Kt/Vs. Covariate adjustment may reduce some of these biases, but as mentioned above the prospect that all such biases can be controlled for is problematic, especially those biases associated with reverse causality. In spite of these limitations, the nephrology community has regarded the accumulated evidence of these observational studies as sufficiently persuasive to increase the national standards for dialysis adequacy, which now stipulate a minimum Kt/V of 1.2 [22]. This illustrates the point that observational results must guide treatment practice in the absence of randomized trials, in spite of their uncertainty. The accumulation of observational results suggesting an association of Kt/V with improved outcome eventually led the NIH to sponsor the HEMO Study, a randomized clinical trial to investigate the effects of even higher doses of dialysis and membrane flux on mortality [7, 8]. The Kt/V arm of the HEMO Study is randomizing patients to either a standard dialysis dose, comparable to a single pool Kt/V of about 1.25, and a high dialysis dose, with a single pool Kt/V of approximately The HEMO Study, which is scheduled for completion in November 2001, should determine whether further increases of dialysis dose beyond the current 1.2 minimum would improve patient outcome based on randomized comparisons which are not subject to the biases described above. Observational Studies and Randomized Trials Blood Purif 2000;18:

6 Concluding Remarks I have argued that observational studies and randomized clinical trials can be viewed as expressions in clinical research of the observational and experimental phases of the scientific method. This perspective clarifies the point that observational studies and randomized clinical trials both play indispensible but complementary roles in clinical research. Furthermore, it is also clear that neither type of study can be effectively replaced by the other and, in particular, that the strength of cause-and-effect inferences provided by randomized clinical trials generally cannot be achieved through observational studies alone. Wolfe and I have pointed out that ethical issues, logistical constraints and resource limitations restrict the application of randomized clinical trials to a comparatively small proportion of biomedical questions. Given finite resources, it is crucial to determine in what circumstances a randomized trial should be conducted and, conversely, in what circumstances satisfactory biomedical decisions can be made from observational research and other sources of evidence without a randomized trial. The requirement for randomized trials is most clear for evaluation of new experimental therapies, where FDA regulations preclude the general use of the therapy until its safety and efficacy are established within the clinical trial setting. On the other hand, randomized trials may not be needed (and may not be ethical) for existing therapies when observational data indicates clear-cut improvements in outcome that are too large to plausibly be explained by the biases described above. The advantage of renal transplantation over artificial dialysis is an example of this scenario. In intermediate cases where the choice of whether to conduct a trial is less clear-cut, it is important to balance several considerations. These include the ability of investigators to ethically randomize patients to each of the therapies under consideration, the cost and feasibility of a trial, the capacity of a trial to demonstrate a beneficial effect on a clinically important outcome with a high statistical power, and the potential impact of the result of a trial on medical practice. References 1 Hempel CG: Philosophy of Natural Science. Englewood Cliffs, Prentice-Hall, Popper KR: The Logic of Scientific Discovery. London, Cambridge University Press, Pocock SJ: Clinical Trials: A Practical Approach. New York, Wiley, Greene T, Lau J, Levey AS: In Neilson EG, Couser WG (eds): Interpretation of Clinical Studies of Renal Disease, from Immunologic Renal Diseases, 1997, chap 40, pp Davey SG, Phillips AN: Confounding in epidemiological studies: Why independent effects may not be all they seem. Br Med J 1992;305: Ellenberg JH: Selection bias in observational and experimental studies. Stat Med 1994;13: Eknoyan G, Levey AS, Beck GJ, Agodoa LY, Daugirdas JT, Kusek JW, Levin NW, Schulman G for the HEMO Study Group: The Hemodialysis (HEMO) Study: Rationale for selection of interventions. Semin Dial 1996;9: HEMO Study Group (prepared by Greene T, Beck GJ, Gassman JJ, Gotch FA, Kusek JW, Levey AS, Levin NW, Schulman G, and Eknoyan G): Design and Statistical Issues in the Hemodialysis (HEMO) Study. Control Clin Trials 2000; in press. 9 Klahr S, Levey AS, Beck GJ, Caggiula AW, Hunsicker L, Kusek JW, Striker G: Modification of Diet in Renal Disease Study Group: The effects of dietary protein restriction and blood pressure control on the progression of renal disease. N Engl J Med 1994;330: Meinert CL: Clinical Trials: Design, Conduct, and Analysis. Oxford, Oxford University Press, Piantadosi S: Clinical Trials: A Methodologic Perspective. New York, Wiley, Davis C: Generalizing from clinical trials. Control Clin Trials 1994;15: Bailed K: Generalizing the results of randomized clinical trials. Control Clin Trials 1994;15: Lowrie EG, Laird NM, Parker TF, Sargent JA: Effect of the hemodialysis prescription on patient morbidity: Report of the National Cooperative Dialysis Study. N Engl J Med 1981;305: Gotch FA, Sargent JA: A mechanistic analysis of the National Cooperative Dialysis Study (NCDS). Kidney Int 1985;28: Hakim RM, Breyer J, Ismail N, Schulman G: Effects of dose of dialysis on morbidity and mortality. Amer J Kidney Dis 1994;23: Collins A, Liao M, Umen A: Urea index (Kt/V) and other predictors of hemodialysis patient survival. Am J Kidney Dis 1994;23: Parker TF, Husni L, Huang W, Lew N, Lowrie EG: Survival of hemodialysis patients in the United States is improved with a greater quantity of dialysis. Am J Kidney Dis 1994;23: Held PJ, Port FK, Wolfe RA, Stannard DC, Carroll CE, Daugirdas JT, Bloembergen WE, Greer JW, Hakim RM: The dose of hemodialysis and patient mortality. Kidney Int 1996;50: Wolfe RA, Ashby VB, Agodoa LYC, Jones CA, Port FK: Body size, dose of hemodialysis and mortality. Am J Kidney Dis 2000;35: Gotch FA, Levin NW, Port FK, Wolfe RA, Uehlinger DE: Clinical outcome relative to the dose of dialysis is not what you think: The fallacy of the mean. Am J Kidney Dis 1997;30: National Kidney Foundation-Dialysis Outcomes Quality Initiative: Clinical practice guidelines for hemodialysis adequacy. Am J Kidney Dis 1997;30(suppl):S22 S Blood Purif 2000;18: Greene