Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

Biases in clinical research Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

Learning objectives Understand goal of measurement and definition of accuracy Describe the threats to causal inferences in clinical studies Understand the role of random variability in clinical studies Describe, understand, and learn how to control the 3 main types of bias: Confounding Information bias / measurement error Selection bias Discuss the concept of generalizability of study results DCR Chapters 4 and 9 2

0 Density.02.04.06.08 Issues with Measurement 50 100 150 200 250 systolic blood pressure, mmhg

Planning measurements: precision and accuracy Measure Measure A F B G Measure C Measure Measure

Measure factors To answer the research question, an epidemiologist needs to measure for each individual: A exposure factors B outcome factors C confounding factors F mediating factors G modifying factors

Goal of measurement Capture the TRUE value of the factor Measurement of factors is often imperfect Imperfect measurement of a Continuous variable measurement error Discrete variable misclassification Note: the precision of clincal measurements maybe very different from the precision of research measurements

Yajnik SC, Yudkin JS. Lancet. 2004;363:163

What do we measure? The factor itself Direct measure Example: body fat mass A surrogate for the factor Indirect measure Example: body mass index Leading to: How well does the surrogate measure the factor? Factors with multiple correlates Example: socioeconomic status

Sources of variability in measuring a factor Within person variability Error due to the measurement tool Examples of measurement tools: bathroom scale; blood pressure cuff (sphygmomanometer); a diet questionnaire; a laboratory assay Error due to the observer Examples of observers: participant, interviewer/observer, abstractor Error in recording the measurement

Error components Measured value = True value + Error Error = Bias + Random Error Systematic component of the error Random component of the error

Bias Systematic difference between the true value and the measured value How close is the measured value to the true value? Synonym for bias: lack of validity Validity on average, the measurement estimates the true measurement

Real example of random error! Body weight, bathroom scale True body weight 180 lbs Inconsistent scale, but set correctly at 0 lbs Moments apart: 1 st measurement: 180.1 lbs 2 nd measurement: 179.4 lbs 3 rd measurement: 180.8 lbs 4 th measurement: 178.2 lbs

Real example of bias! Body weight, bathroom scale True body weight 180 lbs Consistent scale, but fail to set it at 0 lbs: reads -5 lbs Moments apart: 1 st measurement: 175 lbs 2 nd measurement: 175 lbs 3 rd measurement: 175 lbs 4 th measurement: 175 lbs

Measure has random error, but no bias If the measure has no bias, the average of the measured values for the replicates will approach the true value as the number of replicates increases. Distribution of the measured values for the replicates. Body weight Truth 180 lbs

Measure has random error, but no bias The extent of the random error can be small or large (different precision) Synonyms for the extent of random error: Variability Spread Dispersion Scale 1 Scale 2 Body weight Truth 180 lbs

Measure has random error and bias If the measure has bias, the average of the measured values for the replicates will approach the same value, but not the true value, as the number of replicates increases. Body weight Truth Mean of the values of the replicates 180 lbs 185 lbs

Accuracy =lack of bias + high precision High bias Low random error Low bias High random error

The Precision and Accuracy of Measurements Precision Accuracy Definition Best way to assess The degree to which a variable has nearly the same value when measured several times Comparison among repeated measures The degree to which a variable actually represents what it is supposed to represent Comparison with a reference standard Value to study Increase power to detect effects Increase validity of conclusions Threatened by Random error(chance) contributed by The observer The subject The instrument Systematic error(bias) contributed by The observer The subject The instrument

Threats to causal inference Truth in the Universe infer Truth in the Study infer Findings in the study Research Question Random and systematic error Study Plan Random and systematic error Actual Study Target Population Intended Sample Actual subjects Design Implementation Phenomena of interest Intended variables Actual measurement s EXTERNAL VALIDITY INTERNAL VALIDITY 19

Threats to causal inference Lack of precision Random variability - by chance We may observe an association that does not exist or may fail to observe an existing association Lack of internal validity Bias - Systematic errors Confounding Information bias / measurement error Selection bias 20

Threats to causal inference (continued) Incorrect assessment of the direction of causality: We believe that A B But, in reality A B Lack of external validity (generalizability): True effect in the study population But, does not apply to other populations 21

Precision vs. validity Meta-analysis of long-term large randomized controlled trials of statins and coronary heart disease endpoints Cheung BMY, et al. Br J Clin Pharmacol 2004;57:640-51 23

Streptokinase in AMI Meta-analysis Lau J, et al. N Engl J Med 1992;327:248-54 24

Smoking and CVD mortality NHANES II Mortality Study Sample size: 9,205 Length of follow-up: 16 years Prevalence at baseline Current smokers: 32.2% Former smokers: 26.8% Never smokers: 41.0% Hazard ratio for all-cause mortality: Current vs. never smokers: 2.08 (95% CI 1.75 2.48) Former vs. never smokers: 1.32 (95% CI 1.11 1.56) 25

Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality in random samples of N = 500 Ex-smokers Curr. smk [1] 0.7968539 2.0617444 [2] 1.3087417 2.1592132 [3] 1.0062199 1.8700827 [4] 1.2417412 2.0506963 [5] 1.3005539 1.4945079 [6] 2.3954668 3.3830783 [7] 3.3866510 3.4906270 [8] 0.6739165 0.9804119 [9] 1.1980762 2.2224601 [10] 1.1507462 2.3377167 [11] 1.8933153 3.7133280 [12] 0.7163117 1.0573648 [13] 1.4024488 1.5627631 [14] 0.7325482 1.3856760 [15] 1.5731837 1.8970615 [16] 0.9968068 2.5790282 [17] 1.4693343 2.4360451 [18] 0.8029729 1.6165617 [19] 0.9950349 2.0496482 [20] 2.2532537 4.9599239 26

Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing current to never smokers 0.25 0.5 1 2 4 6 8 10 0.25 0.5 1 2 4 6 8 10 N = 500 N = 1,000 0.25 0.5 1 2 4 6 8 10 0.25 0.5 1 2 4 6 8 10 N = 5,000 N = 9,205

Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing former to never smokers 0.25 0.5 1 2 4 6 8 10 0.25 0.5 1 2 4 6 8 10 N = 500 N = 1,000 0.25 0.5 1 2 4 6 8 10 0.25 0.5 1 2 4 6 8 10 N = 5,000 N = 9,205 28

Bias definition Deviation of results or inferences from the truth Any trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically different from the truth Last JM, ed. A dictionary of epidemiology, 4th ed. Oxford, Oxford University Press, 2001 29

Bias classification Many different biases have been described Sackett DL. Bias in analytic research. J Chron Dis 1979;32:51-63 Delgado-Rodriguez M, Llorca J. J Epidemiol Community Health 2004;58:635-41 3 general types of biases: Confounding Misclassification / Information bias Selection bias 30

From causal effect to data Phillips CV. Epidemiology 2003;14:459-66 31

Direction of Bias (I)

Direction of Bias (II)

From causal effect to data

Concepts of confounding MI No MI Coffee 90 60 No coffee 60 90 Odd ratio (OR)= Smokers Nonsmokers MI No MI MI No MI Coffee 80 40 10 20 No Coffee 20 10 40 80 OR in smokers= OR in nonsmokers=

Concepts of confounding Response (R) Exposure (E)

Concepts of confounding Response (R) C is associated with E C causes R CONFOUNDING C = 1 C = 0 Exposure (E)

Concepts of confounding Response (R) C is associated with E C causes R CONFOUNDING C = 1 C = 0 Exposure (E)

Confounding Smith GD, et al. BMJ 1997;315:1641-5 39

Asking about sex Smith GD, et al. BMJ 1997;315:1641-5 40

Comparability of exposure groups Smith GD, et al. BMJ 1997;315:1641-5 41

Sex and mortality Results Smith GD, et al. BMJ 1997;315:1641-5 42

Sex and mortality Recommendations! Smith GD, et al. BMJ 1997;315:1641-5 43

Causal associations Low sex frequency Death from myocardial infarction Low sex frequency Death from myocardial infarction Poor health

Marmor M, et al. Lancet 1982;1:1083-7 45

Hypotheses Marmor M, et al. Lancet 1982;1:1083-7 46

Methods Marmor M, et al. Lancet 1982;1:1083-7 47

Results and interpretation Marmor M, et al. Lancet 1982;1:1083-7 48

Causal diagram Sexual behavior HIV infection AIDS Use of amyl nitrite 49

Confounders are factors that Cause the disease (or are surrogates for causal factors) AND Have a different distribution in exposed and unexposed populations (i.e., are associated with the exposure in the study sample) Both conditions need to be present to have confounding We will also need the additional condition that the confounder is not affected by the exposure

Amyl nitrite, HIV infection, and AIDS Sexual behavior HIV infection AIDS Use of amyl nitrite HIV infection causes AIDS HIV infection and use of amyl nitrite were associated in homosexual men 51

Enrollment and follow-up in HERS Grady D, et al. JAMA 2002;288:49-57 52

Grady D, et al. JAMA 2002;288:49-57 53

54

van Vollenhoven, et al. Lupus 1999;8:181-7 55

56

Jiang R, et al. JAMA 2002;288:2554-60 57

Confounders have to 1. Cause the disease (or be a surrogate measure of a cause) AND 2. Be associated with exposure (i.e., be distributed differently between exposed and unexposed), AND 3. Not affected by exposure (i.e., not be an intermediate variable in the causal pathway) Note: the 3 conditions are necessary for a variable to be a confounder 58

Causal diagram Physical activity, HDL cholesterol, and MI Low physical activity Low HDL cholesterol Myocardial infarction Low physical activity is a cause low HDL cholesterol Low HDL cholesterol is a cause of myocardial infarction HOWEVER, low HDL cholesterol is an intermediate variable in the causal pathway between physical activity and myocardial infarction 59

Uncontrolled confounding Unmeasured confounders Unknown confounders Known confounders that are too expensive or difficult to measure Residual confounding Confounder is measured imperfectly, and cannot be controlled completely 61

Results and interpretation Marmor M, et al. Lancet 1982;1:1083-7 62

Confounding by indication Hak E, et al. J Epidemiol Community Health 2002;56:951-5 63

In practice (I) Prior knowledge on the biological and other causal relationships is needed to properly identify which variables to adjust for Do NOT apply statistical criteria to decide if the conditions for confounding are present Testing for the association of confounder with exposure and of confounder with disease Stepwise selection procedures Consider if exposed and unexposed subjects are comparable with respect to their risk of disease (except for exposure) 64

In practice (II) Consider which determinants of disease may be responsible for the lack of comparability Elaborate causal diagram Identify causal factors that may be different between exposed and unexposed Obtain information on potential confounders Measuring confounders with error will result in residual confounding after adjustment Use statistical techniques to adjust for potential confounders 65

Methods to control for confounding In the design of the study Randomization Restriction Matching primarily in case-control studies In the analysis Standardization Stratification Multivariate models Propensity scores Inverse probability weighting Sensitivity analysis 66

Restriction Restrict eligibility into the study to one category of the confounder Advantages Convenient, inexpensive and easy to implement Adequate control of confounder Simple analysis Disadvantages Cannot evaluate effect modification May limit generalizability For continuous variables, need to group the restriction variable (possibility of residual confounding) Can only be applied to a small number of variables 67

Restriction to lifetime non-smokers to avoid confounding by smoking

Kabat GC, et al. Cancer 1986;57:362-7 69

From causal effect to data Phillips CV. Epidemiology 2003;14:459-66 70

Selection bias The measure of association observed in the study sample is different to the measure of association in the source population Selection into the study is affected both by the exposure (or by a cause of the exposure) AND by a cause of the outcome (in cohort studies) or by the outcome (in case-control studies) Source population Study population Exposed Disease No disease Disease No disease A B Exposed b a C D Nonexposed Nonexposed c d 71

폭로군 (1000 명 ) 비폭로군 (2000 명 ) Risk = 10 1000 = 0.01 Risk = 5 2000 = 0.0025 폭로군에서 risk 0.01 Relative risk = = 비폭로군에서 risk 0.0025 = 4

폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 대조군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = 10 5 10 5 = 2 = 2 Odds ratio = 2 2 = 1

예 1) Fat intake 은대장암의위험요인인가? 환자군 : multi-center 대장암신환 대조군 : 대장암이없는종검수진자

폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 대조군 : 15 명 ( 폭로 Hx 에서 1 명, 비폭로 Hx 에서 14 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = 10 5 1 14 = 2 = 0.0714 Odds ratio = 2 0.0714 = 28

예 2) 흡연은방광암의위험요인인가? 환자군 : multi-center 방광암신환 대조군 : 방광암이없는종검수진자

폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로군에서 10 명, 비폭로군에서 5 명 ) 대조군 : 15 명 ( 폭로군에서 5 명, 비폭로군에서 10 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = 10 5 5 10 = 2 = 0.5 Odds ratio = 2 0.5 = 4

Selection bias in Case-Control Study

커피음용비율 (%) 60 50 40 30 20 10 0 환자군대조군일반인

Selection bias in Case-Control Study

Selection bias in cohort studies Immigrative selection bias Selection into the cohort is affected both by exposure (or by a cause of exposure) and by risk of disease Emigrative selection bias Selection out of the cohort (losses to follow-up) are affected both by exposure (or by a cause of exposure) and by risk of disease 85

86

Honolulu Heart Study Rate of CHD, stroke and total mortality in 11,136 men of Japanese ancestry eligible in 1965 8,006 men examined and 3,130 not examined (60% completed a mailed questionnaire) Baseline smoking as current, ever or never All 11,136 followed through 1982 87

Differential smoking by participation Examined Unexamined N = 8,006 N = 3,130 Smoking: Ever 69% 72% Current 48% 53% Age-standardized %, p value < 0.01 Selection into cohort associated with smoking 88

Non-participants Participants Benfante R, et al. Am J Epidemiol 1989;130:1088-100 89

Rate ratios for current smoking Examined (Study) N = 8,006 Un-Examined N = 3,130 Source N = 11,136 Total 1.63 1.36 1.58 CVD 1.71 1.16 1.59 Stroke 1.92 0.95 1.69 90

Grady D et al, JAMA 2002;288:49-57

Grady D et al, JAMA 2002;288:49-57

CHD events since randomization HERS and HERS II Grady D, et al. JAMA 2002;288:49-57 93

Psaty BM, et al. JAMA 2001;285:906-913

Psaty BM, et al. JAMA 2001;285:906-913

Prothrombin variant 20210 G A, HRT, and non-fatal MI Population based case-control study Psaty BM, et al. JAMA 2001;285:906-13 96

Selection bias in cohort studies of HRT and CVD Postmenopausal women with prothrombotic mutations may die or have CVD early after onset of treatment These women would not be included in cohort studies of HRT and CVD Source population Exposed Disease No disease Study population Disease No disease A B Exposed b a C D Nonexposed Nonexposed c d 97

Selection bias in cohort studies of HRT and CVD HRT Selection Outcome Prothrombotic mutations

Psaty BM, et al. JAMA 2001;285:906-13 99

Samaha FF, et al. N Engl J Med 2003;348:2074-81 100

Samaha FF, et al. N Engl J Med 2003;348:2074-81 101

102

Selection bias due to losses of follow-up in RCT of Atkins diet Assigned diet Selection Outcome (weight loss) Age, other factors

Minimizing selection bias Random sampling from source population Limit losses to follow up Sensitivity analysis 104

Healthy Worker Effect

Dibbs E, et al. Circulation 1982;65:943-6 106

107

From causal effect to data Phillips CV. Epidemiology 2003;14:459-66 108

Measurement error can affect Exposure Outcome Confounders Mediators Modifying factors 109

Error components Measured value = True value + Error Error = Bias + Random Error Systematic component of the error Random component of the error

Quantification of measurement error Dichotomous variables Sensitivity, specificity Kappa statistic Categorical variables Spearman correlation coefficient Kappa statistic Continuous variables Coefficient of variation Intraclass correlation coefficient (reliability coefficient) 111

Differential vs. non-differential errors Non-differential measurement error Measurement error in the variable in question (e.g., the exposure) does not depend on the levels of other variables (e.g., the outcome, confounders, etc) Differential measurement error Measurement error depends on the levels of other variables (for instance, when sensitivity and specificity for measuring disease are different in exposed and unexposed participants) 112

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 100% Observed RR = 1.71 Yes Diseased No Exposed Yes 160 640 800 No 100+40 900+160 1000+200 300 1700 2000 113

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 100% Specificity = 90% Observed RR = 1.91 Yes Diseased No Exposed Yes 200+10 800+90 1000+100 No 90 810 900 300 1700 2000 114

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 90% Observed RR = 1.60 Yes Diseased No Exposed Yes 160+10 640+90 800+100 No 90+40 810+160 900+200 300 1700 2000 115

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure In this case, measurement error will induce a bias will be towards the null, unless The test is uninformative or misleading The true effect is null The magnitude of the bias depends on: Sensitivity and specificity The prevalence of the exposure The risk of the disease The magnitude of the true effect The measure of association used 116

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure 117

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

Non-differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED DISEASE Sensitivity = 80% Specificity = 90% Observed RR = 1.41 Yes Diseased No Exposed Yes 160+80 720+40 1000 No 80+90 810+20 1000 240+170 1530+60 2000 121

Regression towards the mean When a variable is measured with random error and we select participants with observed extreme values, their true underlying values are on average closer to the population mean Measure BP and select participants with SBP > 140 mmhg Consequences Inconsistencies in diagnosis and classification Biases in evaluation of interventions Inefficiency in planning studies 129

Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED Yes DISEASE Sens in exposed = 90% Sens in unexposed = 80% Spec in exposed = 100% Spec in unexposed = 100% Observed RR = 2.25 Diseased No Exposed Yes 180 800+20 1000 No 80 900+20 1000 260 1700+40 2000 130

Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes 200 800 1000 No 100 900 1000 300 1700 2000 TABLE WITH MISCLASSIFIED Yes DISEASE Sens in exposed = 100% Sens in unexposed = 100% Spec in exposed = 90% Spec in unexposed = 80% Observed RR = 1.00 Diseased No Exposed Yes 200+80 720 1000 No 100+180 720 1000 300+260 1440 2000 131

Differential measurement error Can bias measures of association in any direction The magnitude can be substantial, even with small differences in sensitivity or specificity In cohort studies, an important concern is differential classification of disease as a function of exposure Diagnostic bias Surveillance bias Mask follow-up procedures and outcome assessment 132

Main points on measurement error Measurement errors are pervasive in epidemiological studies Non-differential, independent errors in exposure or outcome tend to bias associations towards the null, but there are exceptions Differential or dependent errors can bias the association in either direction If sensitivity / specificity or ICC are known from validation studies, we can correct the measures of association 133

Strategies for increasing accuracy of measurements Standardize measurement methods in an operations manual Train and certify observers Refine the instruments Automate instruments and procedures Calibrate equipment Make unobtrusive measurements Blind measurements Take repeated measurements

Generalizability (external validity) 135

PPV Information Sheet 136

Generalizability is a judgment Consider if the same biological / social mechanisms apply in the target population as in the source population Consider if the prevalence of factors that may modify the effect of the exposure are different in the target and in the source population E.g., genetic determinants Be careful 137

Any question? Sh703.yoo@gmail.com