INTERNAL VALIDITY, BIAS AND CONFOUNDING

OCW Epidemiology and Biostatistics, 2010 J. Forrester, PhD Tufts University School of Medicine October 6, 2010 INTERNAL VALIDITY, BIAS AND CONFOUNDING Learning objectives for this session: 1) Understand and distinguish the concepts of internal and external validity 2) Understand what is meant by the term bias 3) Distinguish bias from imprecision 4) Understand what is meant by the term confounding 5) Know the characteristics of a confounding variable 6) Understand what study design features are used to avoid confounding 7) Distinguish confounding and effect modification (interaction) Outside preparation: Gordis Chapter 15, pages 251-261 Internal validity is the degree to which the study findings represent a true reflection of the exposure-outcome association in the target population. (e.g. How close the estimated relative risk in a study is to the true, but unknown relative risk) There are three steps to assess the internal validity of a study: 1) Rule out bias 2) Rule out confounding (a special kind of bias) 3) Rule out chance (by means of statistical tests which we will see later) What is bias? Bias is a distortion of the results of a study such that the results do not reflect the true (but unknown) exposure-outcome association. More specifically bias is a systematic error in the measurement of an association between two variables. 1

Bias vs. imprecision Bias can be thought of as the distance an observation is from truth. Precision can be thought of as reproducibility. Imprecision is lack of reproducibility. Each one of the s below can be thought of as a study result. The bull's eye in the target is the true exposure - outcome association that we are attempting to estimate (e.g. the true RR). No Bias & No Imprecision Bias & No Imprecision No bias & Imprecision Bias & Imprecision (Here, the average of the s will fall On the bull s eye) 2

There is no coherent taxonomy of biases. The same bias can have more than one name. Different textbooks classify biases in different ways. We use the classification below. There are three major types of bias. Most biases fall under one of these three types: 1) Confounding 2) Selection bias (also called "sample distortion bias") 3) Information or observation bias: There are three major types of information bias: 1) Random (non-differential) misclassification of exposure/outcome 2) Non-Random (differential) misclassification of exposure/outcome 3) Loss-to-follow-up bias Confounding The presence of confounding induces bias and is a threat to a study s internal validity. A confounder is a risk factor for the disease that is not the exposure under study and that is not evenly distributed among the exposure groups. Our goal is to compare groups that are alike with respect to all risk factors for the outcome except the exposure we are studying. "Imbalance" in risk factors among the exposure groups will result in an unfair comparison of groups because the group with more risk factors is bound to develop more disease. This imbalance in risk factors across study arms, regardless of the study design, is called confounding. Characteristics of a confounder: To be a confounder a variable must fulfill three conditions: 1) A confounder must be a risk factor for the outcome. Only an imbalance in risk factors for the outcome of interest can lead to confounding. We do not care about imbalance in factors that are not risk factors for the disease. For example, we do not care about the proportion of Red Sox fans in each group since being a Red Sox fan, as far as we know, is only a risk factor for disappointment. 2) A confounder must be associated with the study exposure in the study. The ways in which a confounder can be associated with the exposure in a study are as follows: In a RCT: Treatment and placebo groups are not balanced for disease risk factors after randomization (most often a problem in studies with small numbers of subjects). If the risk factor is balanced across the treatment and placebo groups then the risk factor is not a confounder because it is not associated with the exposure. 3

In an observational study: Exposed and non-exposed groups are not balanced for other disease risk factors once the groups are assembled. If the risk factor is balanced across the exposure groups then the risk factor is not a confounder because it is not associated with the exposure. 3) A confounder must not be an intermediate risk factor (a.k.a. a mediator) in the causal pathway between the exposure and outcome. Cholesterol is an example of an intermediate risk factor or mediator of the association between a high fat diet and cardiovascular disease: High Fat Diet Cholesterol Cardiovascular disease We will see more on mediators later. Methods to prevent and control confounding (i.e. to force the exposure groups to be balanced for disease risk factors) 1) Control of confounding in the study design: Randomization: Randomization helps to balance the risk factors across the study arms, especially if a large number of subjects are randomized. If the study is small (about 100 subjects or less would be considered a small RCT), randomization helps but does not guarantee balance. Restriction: You can control confounding by restricting the study sample to one level of the confounder. For example, if you are worried about confounding by gender you could choose to study only women. Clinical trials will often mention exclusion criteria that are motivated by a desire to reduce confounding. For example, it would be reasonable in a trial testing the effect of a new cholesterol-lowering drug on the incidence of cardiovascular disease to exclude persons who have been diagnosed with cancer because there are so many aspects of cancer that could confound a study of cardiovascular disease. While this strategy may eliminate confounding by cancer, it limits the generalizability of the study. You may want to treat high cholesterol in your patients with the new drug described above, including those patients who have had a diagnosis of cancer in their past. The restriction of the trial to persons without a cancer diagnosis may limit your ability to judge how the medication will act in these people. Matching: This method is used in observational study designs (not in RCTs). In a cohort study matching makes intuitive sense. If you want to control confounding by gender you recruit one exposed woman and match her to an unexposed woman, followed by one exposed man and an unexposed man, etc. Matching can be difficult if you have many variables to match for since it will be harder to find pairs of subjects with the same risk factor profile except the exposure under study. Matching is both expensive and inefficient because it is difficult and time consuming to recruit study subjects and you have to throw away recruited subjects for whom you have no match. In a case-control study matching is not intuitive. Suffice it to say that 4

matching cases and controls for important confounders helps us to make a more efficient adjustment for the confounders in the statistical analysis (described below). 2) Control of confounding in the statistical analyses: Stratification: This is done by calculating the measure of association (e.g. RR, OR) within the confounder groups. For example, if the confounder is gender, calculate the RR in women and, separately, calculate the RR in men. Each of these RRs is free of confounding by gender. If there is no evidence that the RRs differ in magnitude, you can collapse these two RRs into one using a weighted average of the RR in women and the RR in men. This average RR is free of confounding by gender. This was the old fashioned method used by epidemiologists before everyone had a PC with statistical software. It still works and it is a method used often by epidemiologists and statisticians to look at data before it is given to the computer to calculate. Multivariate Analysis: This is a fancy form of stratification. It is fancy because the computer can stratify by many variables simultaneously using multiple regression techniques, which will simultaneously adjust for many potential confounders, which we will see later. When reading the medical literature, look for the words adjustment or control for potential confounders in the statistical section or in the footnotes to the tables to see if the authors report adjusting for any imbalance in the groups being compared. A well-written article will list all of the factors that were included in the adjustment. How to recognize confounding Let s use the evaluation of alcohol as a confounder of the relationship between smoking (the exposure) and myocardial infarction (the outcome) in a study with 100 subjects. You suspect that alcohol could be a confounder because your preliminary analyses showed that the smokers were more likely to drink alcohol than the non-smokers (i.e. the exposure groups were imbalanced on drinking). Alcohol, in moderate amounts, is known to protect against cardiovascular disease. Step 1) Calculate the RR the usual way with drinkers and non-drinkers mixed all together: You find the overall "crude" RR of MI & smoking = 3.0 Is alcohol a confounder? In other words, is 3.0 the true, unbiased RR? 5

Step 2) Divide the data into drinkers and non-drinkers and calculate the RR within drinkers and the RR within non-drinkers (say half of the 100 subjects are drinkers): In the 50 Drinkers: In the 50 Non-drinkers: RR of MI & Smoking = 3.9 RR of MI & Smoking = 4.1 Each of these two RRs is free of confounding by alcohol use because the smoking-mi association was evaluated separately within groups that drink or do not drink. Step 3) To calculate the overall unconfounded ( adjusted ) RR, take the average of the RR in drinkers and non-drinkers: Adjusted RR = (3.9 + 4.1) / 2 = 4.0 (This adjusted RR applies to all 100 subjects) Our data demonstrate evidence that alcohol was confounding the association between smoking and MI because the crude RR (3.0) is quite different from the average (adjusted) RR calculated drinkers and non-drinkers (4.0). How different is different is a judgment call. Because drinking was providing some protection from MI to the smokers, when the effect of drinking was removed, the association of smoking with MI was even bigger an unadjusted (crude) RR of 3.0 became an adjusted RR of 4.0. The evidence of confounding: Crude RR of 3.0 not equal to adjusted RR of 4.0 To assess confounding: Compare the crude RR (or OR) to the adjusted RR (or OR) In a journal article, it is unusual to see the data laid out as above. When reading an article, to evaluate confounding, you should look for differences in the RR between the unadjusted analysis (the crude RR), and the RR following multivariable adjustment (the "adjusted" RR). This is usually shown in a table format. If the unadjusted and adjusted RRs differ by a lot, this indicates that at least one of the factors that was adjusted for was, indeed, a confounder. Regardless of the size of the difference between the crude and adjusted RR, the adjusted RR is always the most valid estimate of the true association between the exposure and outcome. For an example of this look at Table 2 of the article by Tsutsumi et al. on occupational stress and the risk of stroke (the assigned reading for small group #2). The authors present RRs comparing the risk of total stroke in men in high-strain jobs vs. men in low strain jobs (the reference group) adjusted for age and area. The crude RR is 28 /8808 divided by 7/5311 = 2.41. Adjustment for age and area gives a RR of 2.62. This is a slightly different value indicating some confounding by age and area. The value 2.62 is the more valid of the two RRs. 6

Table 2. Associations of Psychosocial Job Characteristics With Incident Stroke Hazard Ratio Men (N=3190) Person-years No. of Events Adjusted for Age and Area Total stroke Low-strain job 5311 7 1 (Reference) High-strain job 8808 28 2.62 From Tsutsumi et al. Prospective study on occupational Stress and Risk of Stroke Arch Intern Med 2009; 169: 56-61. Other comments about confounding: Confounding is the only kind of bias that can be corrected easily after the study data are collected. However, you can only adjust (control) for risk factors you have measured. If you did not collect data on a particular risk factor, you will not know if the comparison groups are balanced with respect to that risk factor and you will not be able remove the factor s possible confounding influence by stratification or multivariate adjustment. Confounding that remains in a study after adjustment is called residual confounding If you measure a confounder using a poor quality instrument such that there is misclassification in the determination of the confounder, you will still have residual confounding after adjustment. For example, if alcohol is the potential confounder of concern and the data were not carefully collected or the subjects were not truthful in their response about their drinking habits there may be residual confounding after adjustment for alcohol in the analyses because of the poor capture of the alcohol data. Control for an intermediate risk factor (a mediator) in the causal pathway, will wash out the association of the exposure with the outcome and thereby underestimate the true exposure-disease relationship. For example, High Fat Diet Cholesterol Cardiovascular disease In a study of the relation between a high fat diet and cardiovascular disease (CVD) adjustment for cholesterol ( a mediator) will wash out the association between a high fat diet and CVD because higher cholesterol is one of the ways in which a high fat diet leads to CVD. Epidemiologists call this over adjusting or over controlling. Of course, this assumes something is known about the biologic pathway between a high fat diet and CVD. Often we don t know the pathway. In the cardiovascular literature it is now common to see analyses that include not just confounders but also mediators. The motive for examining mediators in the analysis is to explore the biologic pathways. Using the example above, say we did not know how a high fat diet increases the risk of CVD. We could explore the hypothesis that one of the mechanisms is by increasing serum cholesterol levels. We would adjust for cholesterol and find that the RR describing the high fat diet - CVD association is attenuated. This would suggest that the mechanism by which a high fat diet increases the risk of CVD works through an elevation in cholesterol. To see an example of an exploration for mediators look again at Table 2 of the article by Tsutsumi et al. on occupational stress and the risk of stroke. In the far right column of 7

Table 2, the authors present their data on the association of occupational stress and stroke adjusted for biologic risk factors in addition to age, area, socio-demographic and behavioral risk factors. They explored the extent to which the elevated risk of stroke in persons with high-strain jobs was mediated by its effect on biologic risk factors. As they noted in the text, the RR adjusted for age, area, socio-demographic and behavioral risk factors alone (RR of total stroke=2.73 in the men) was attenuated to a value of 2.53 by adjusting for biologic factors. They write in the comment section of the paper, "We found that adjustments for biologic risk factors slightly attenuated the association between job strain and an incident stroke. These findings suggest that the association between job strain and cardiovascular diseases is mediated by the presence of 1 or more chronic diseases, such as obesity, hypertension, glucose intolerance, and dyslipidemia. However, adjustments for such variables did not fully account for the associations between job strain and stroke." Large randomized controlled trials do not suffer from confounding because the randomization of large numbers of study participants balances the risk factor profile across the study arms. Being balanced, the risk factors for the outcome do not satisfy criterion #2 for confounding. Going back to the low-dose aspirin study by Ridker et al. you will see in Table 1 of the trial results that the risk factor profile of the aspirin group is almost identical to the risk factor profile of the placebo group the magic of randomization! Therefore, confounding is not a problem in this study and no adjustments were necessary. Confounding vs. Effect Modification While an assessment of confounding involves comparing the crude and adjusted RRs, effect modification is evaluated by comparing RR in sub-groups based on some third variable. To do this, you compare the RR in subgroup 1 with the RR in subgroup 2 (Here, the crude RR is irrelevant). For example, in the low-dose aspirin study by Ridker et al. the RR describing the association of low-dose aspirin with cardiovascular disease was evaluated in women of different age groups. As we saw, low-dose aspirin was most effective (lowest RR) in women > 65 years of age. We concluded that age was a modifier of the effect of low-dose aspirin on the association of low-dose aspirin and cardiovascular disease. Since the study was a large RCT, there was no concern for confounding. In the example of smoking as a risk factor for myocardial infarction, used above, you should compare the RR for smoking and MI in the drinkers vs. the RR for smoking and MI in the non-drinkers i.e. 3.9 vs. 4.1. If they are very different, then alcohol is an effect modifier. In this example, the RRs are not very different, so alcohol is not an effect modifier. However, note that alcohol was a confounder because the crude and adjusted RRs were quite different from each other (4.0 vs. 3.0). To assess confounding: Compare crude to the adjusted RR. If they are very different (by the eye-ball test ), there was confounding To assess effect modification: Compare subgroup RRs to each other. If they are very different, there is effect modification by the stratification variable 8

A confounder must be a risk factor for the disease. An effect modifier does not need to be a risk factor for the disease - just some other variable that is biologically related to the disease process. However, effect modifiers are often risk factors for the disease. External validity: Another important aspect to consider when evaluating a study is its external validity. External validity is the degree to which study findings can be generalized beyond the study target population, - i.e., to whom do the results of this study apply? For example, can you generalize a study done in young to middle aged Caucasian adults to minorities, children, or the elderly? Can you generalize the results of a study done in normal weight adults to obese adults? Can you generalize a study of aspirin and heart disease done in men to women? The question of generalizability requires common sense. Is there reason to believe that the biologic processes examined in the current study are the same or different in another group of individuals? Use your head scientists generalize from mice to men every day. However, sometimes generalization it is not appropriate. Confounding is an issue of internal validity, while effect modification is an issue of external validity. You will never see confounding or effect modification laid out in a paper as below. These numbers are given simply as a guide to the evaluation of confounding and effect modification. Example of confounding (with no effect modification) by hair color in a case-control study of eye color and risk of lupus (fictitious numbers: eyes and hair color are not known risk factors for lupus, but race - African American or Hispanic - is a known risk factor) Overall crude results: OR = 1.6 Brown eyes 257 200 Blue eyes 243 300 Hair color dark: OR= 1.0 Hair color light: OR= 1.0 Brown eyes 57 100 Blue eyes 143 250 Brown eyes 200 100 Blue eyes 100 50 Overall adjusted OR = 1.0 (550/1000 *1.0 + 450/1000*1.0) the weighted average of stratum-specific ORs). Note the ORs in the dark and light hair are identical, indicating no 9

effect modification. Interpretation: Hair color was confounding the association of eye color and the risk of lupus. Once hair color was adjusted, there was no association between eye color and lupus. Example of effect modification by hair color in a case-control study of eye color and risk of lupus (fictitious numbers: eyes and hair color are not known risk factors for lupus, but race - African American or Hispanic - is a known risk factor) Overall results: OR = 2.3 Brown eyes 32 168 Blue eyes 15 185 Hair color dark: OR= 4.7 Brown eyes 18 42 Blue eyes 5 55 Hair color light: OR= 1.4 Brown eyes 14 126 Blue eyes 10 130 Interpretation: The effect of eye color on the risk of lupus depends on hair color. The risk associated with having brown eyes is greater if you also have dark hair. Hair color modifies the association of eye color and the risk of lupus. 10