Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

Transcription

1 Biases in clinical research Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

2 Learning objectives Understand goal of measurement and definition of accuracy Describe the threats to causal inferences in clinical studies Understand the role of random variability in clinical studies Describe, understand, and learn how to control the 3 main types of bias: Confounding Information bias / measurement error Selection bias Discuss the concept of generalizability of study results DCR Chapters 4 and 9 2

3 0 Density Issues with Measurement systolic blood pressure, mmhg

4 Planning measurements: precision and accuracy Measure Measure A F B G Measure C Measure Measure

5 Measure factors To answer the research question, an epidemiologist needs to measure for each individual: A exposure factors B outcome factors C confounding factors F mediating factors G modifying factors

6 Goal of measurement Capture the TRUE value of the factor Measurement of factors is often imperfect Imperfect measurement of a Continuous variable measurement error Discrete variable misclassification Note: the precision of clincal measurements maybe very different from the precision of research measurements

7 Yajnik SC, Yudkin JS. Lancet. 2004;363:163

8 What do we measure? The factor itself Direct measure Example: body fat mass A surrogate for the factor Indirect measure Example: body mass index Leading to: How well does the surrogate measure the factor? Factors with multiple correlates Example: socioeconomic status

9 Sources of variability in measuring a factor Within person variability Error due to the measurement tool Examples of measurement tools: bathroom scale; blood pressure cuff (sphygmomanometer); a diet questionnaire; a laboratory assay Error due to the observer Examples of observers: participant, interviewer/observer, abstractor Error in recording the measurement

10 Error components Measured value = True value + Error Error = Bias + Random Error Systematic component of the error Random component of the error

11 Bias Systematic difference between the true value and the measured value How close is the measured value to the true value? Synonym for bias: lack of validity Validity on average, the measurement estimates the true measurement

12 Real example of random error! Body weight, bathroom scale True body weight 180 lbs Inconsistent scale, but set correctly at 0 lbs Moments apart: 1 st measurement: lbs 2 nd measurement: lbs 3 rd measurement: lbs 4 th measurement: lbs

13 Real example of bias! Body weight, bathroom scale True body weight 180 lbs Consistent scale, but fail to set it at 0 lbs: reads -5 lbs Moments apart: 1 st measurement: 175 lbs 2 nd measurement: 175 lbs 3 rd measurement: 175 lbs 4 th measurement: 175 lbs

14 Measure has random error, but no bias If the measure has no bias, the average of the measured values for the replicates will approach the true value as the number of replicates increases. Distribution of the measured values for the replicates. Body weight Truth 180 lbs

15 Measure has random error, but no bias The extent of the random error can be small or large (different precision) Synonyms for the extent of random error: Variability Spread Dispersion Scale 1 Scale 2 Body weight Truth 180 lbs

16 Measure has random error and bias If the measure has bias, the average of the measured values for the replicates will approach the same value, but not the true value, as the number of replicates increases. Body weight Truth Mean of the values of the replicates 180 lbs 185 lbs

17 Accuracy =lack of bias + high precision High bias Low random error Low bias High random error

18 The Precision and Accuracy of Measurements Precision Accuracy Definition Best way to assess The degree to which a variable has nearly the same value when measured several times Comparison among repeated measures The degree to which a variable actually represents what it is supposed to represent Comparison with a reference standard Value to study Increase power to detect effects Increase validity of conclusions Threatened by Random error(chance) contributed by The observer The subject The instrument Systematic error(bias) contributed by The observer The subject The instrument

19 Threats to causal inference Truth in the Universe infer Truth in the Study infer Findings in the study Research Question Random and systematic error Study Plan Random and systematic error Actual Study Target Population Intended Sample Actual subjects Design Implementation Phenomena of interest Intended variables Actual measurement s EXTERNAL VALIDITY INTERNAL VALIDITY 19

20 Threats to causal inference Lack of precision Random variability - by chance We may observe an association that does not exist or may fail to observe an existing association Lack of internal validity Bias - Systematic errors Confounding Information bias / measurement error Selection bias 20

21 Threats to causal inference (continued) Incorrect assessment of the direction of causality: We believe that A B But, in reality A B Lack of external validity (generalizability): True effect in the study population But, does not apply to other populations 21

22

23 Precision vs. validity Meta-analysis of long-term large randomized controlled trials of statins and coronary heart disease endpoints Cheung BMY, et al. Br J Clin Pharmacol 2004;57:

24 Streptokinase in AMI Meta-analysis Lau J, et al. N Engl J Med 1992;327:

25 Smoking and CVD mortality NHANES II Mortality Study Sample size: 9,205 Length of follow-up: 16 years Prevalence at baseline Current smokers: 32.2% Former smokers: 26.8% Never smokers: 41.0% Hazard ratio for all-cause mortality: Current vs. never smokers: 2.08 (95% CI ) Former vs. never smokers: 1.32 (95% CI ) 25

26 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality in random samples of N = 500 Ex-smokers Curr. smk [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

27 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing current to never smokers N = 500 N = 1, N = 5,000 N = 9,205

28 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing former to never smokers N = 500 N = 1, N = 5,000 N = 9,205 28

29 Bias definition Deviation of results or inferences from the truth Any trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically different from the truth Last JM, ed. A dictionary of epidemiology, 4th ed. Oxford, Oxford University Press,

30 Bias classification Many different biases have been described Sackett DL. Bias in analytic research. J Chron Dis 1979;32:51-63 Delgado-Rodriguez M, Llorca J. J Epidemiol Community Health 2004;58: general types of biases: Confounding Misclassification / Information bias Selection bias 30

31 From causal effect to data Phillips CV. Epidemiology 2003;14:

32 Direction of Bias (I)

33 Direction of Bias (II)

34 From causal effect to data

35 Concepts of confounding MI No MI Coffee No coffee Odd ratio (OR)= Smokers Nonsmokers MI No MI MI No MI Coffee No Coffee OR in smokers= OR in nonsmokers=

36 Concepts of confounding Response (R) Exposure (E)

37 Concepts of confounding Response (R) C is associated with E C causes R CONFOUNDING C = 1 C = 0 Exposure (E)

38 Concepts of confounding Response (R) C is associated with E C causes R CONFOUNDING C = 1 C = 0 Exposure (E)

39 Confounding Smith GD, et al. BMJ 1997;315:

40 Asking about sex Smith GD, et al. BMJ 1997;315:

41 Comparability of exposure groups Smith GD, et al. BMJ 1997;315:

42 Sex and mortality Results Smith GD, et al. BMJ 1997;315:

43 Sex and mortality Recommendations! Smith GD, et al. BMJ 1997;315:

44 Causal associations Low sex frequency Death from myocardial infarction Low sex frequency Death from myocardial infarction Poor health

45 Marmor M, et al. Lancet 1982;1:

46 Hypotheses Marmor M, et al. Lancet 1982;1:

47 Methods Marmor M, et al. Lancet 1982;1:

48 Results and interpretation Marmor M, et al. Lancet 1982;1:

49 Causal diagram Sexual behavior HIV infection AIDS Use of amyl nitrite 49

50 Confounders are factors that Cause the disease (or are surrogates for causal factors) AND Have a different distribution in exposed and unexposed populations (i.e., are associated with the exposure in the study sample) Both conditions need to be present to have confounding We will also need the additional condition that the confounder is not affected by the exposure

51 Amyl nitrite, HIV infection, and AIDS Sexual behavior HIV infection AIDS Use of amyl nitrite HIV infection causes AIDS HIV infection and use of amyl nitrite were associated in homosexual men 51

52 Enrollment and follow-up in HERS Grady D, et al. JAMA 2002;288:

53 Grady D, et al. JAMA 2002;288:

54 54

55 van Vollenhoven, et al. Lupus 1999;8:

56 56

57 Jiang R, et al. JAMA 2002;288:

58 Confounders have to 1. Cause the disease (or be a surrogate measure of a cause) AND 2. Be associated with exposure (i.e., be distributed differently between exposed and unexposed), AND 3. Not affected by exposure (i.e., not be an intermediate variable in the causal pathway) Note: the 3 conditions are necessary for a variable to be a confounder 58

59 Causal diagram Physical activity, HDL cholesterol, and MI Low physical activity Low HDL cholesterol Myocardial infarction Low physical activity is a cause low HDL cholesterol Low HDL cholesterol is a cause of myocardial infarction HOWEVER, low HDL cholesterol is an intermediate variable in the causal pathway between physical activity and myocardial infarction 59

60

61 Uncontrolled confounding Unmeasured confounders Unknown confounders Known confounders that are too expensive or difficult to measure Residual confounding Confounder is measured imperfectly, and cannot be controlled completely 61

62 Results and interpretation Marmor M, et al. Lancet 1982;1:

63 Confounding by indication Hak E, et al. J Epidemiol Community Health 2002;56:

64 In practice (I) Prior knowledge on the biological and other causal relationships is needed to properly identify which variables to adjust for Do NOT apply statistical criteria to decide if the conditions for confounding are present Testing for the association of confounder with exposure and of confounder with disease Stepwise selection procedures Consider if exposed and unexposed subjects are comparable with respect to their risk of disease (except for exposure) 64

65 In practice (II) Consider which determinants of disease may be responsible for the lack of comparability Elaborate causal diagram Identify causal factors that may be different between exposed and unexposed Obtain information on potential confounders Measuring confounders with error will result in residual confounding after adjustment Use statistical techniques to adjust for potential confounders 65

66 Methods to control for confounding In the design of the study Randomization Restriction Matching primarily in case-control studies In the analysis Standardization Stratification Multivariate models Propensity scores Inverse probability weighting Sensitivity analysis 66

67 Restriction Restrict eligibility into the study to one category of the confounder Advantages Convenient, inexpensive and easy to implement Adequate control of confounder Simple analysis Disadvantages Cannot evaluate effect modification May limit generalizability For continuous variables, need to group the restriction variable (possibility of residual confounding) Can only be applied to a small number of variables 67

68 Restriction to lifetime non-smokers to avoid confounding by smoking

69 Kabat GC, et al. Cancer 1986;57:

70 From causal effect to data Phillips CV. Epidemiology 2003;14:

71 Selection bias The measure of association observed in the study sample is different to the measure of association in the source population Selection into the study is affected both by the exposure (or by a cause of the exposure) AND by a cause of the outcome (in cohort studies) or by the outcome (in case-control studies) Source population Study population Exposed Disease No disease Disease No disease A B Exposed b a C D Nonexposed Nonexposed c d 71

72 폭로군 (1000 명 ) 비폭로군 (2000 명 ) Risk = = 0.01 Risk = = 폭로군에서 risk 0.01 Relative risk = = 비폭로군에서 risk = 4

73 폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 대조군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = = 2 = 2 Odds ratio = 2 2 = 1

74 예 1) Fat intake 은대장암의위험요인인가? 환자군 : multi-center 대장암신환 대조군 : 대장암이없는종검수진자

75 폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로 Hx 에서 10 명, 비폭로 Hx 에서 5 명 ) 대조군 : 15 명 ( 폭로 Hx 에서 1 명, 비폭로 Hx 에서 14 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = = 2 = Odds ratio = = 28

76 예 2) 흡연은방광암의위험요인인가? 환자군 : multi-center 방광암신환 대조군 : 방광암이없는종검수진자

77 폭로군 (1000 명 ) 비폭로군 (2000 명 ) 환자군 : 15 명 ( 폭로군에서 10 명, 비폭로군에서 5 명 ) 대조군 : 15 명 ( 폭로군에서 5 명, 비폭로군에서 10 명 ) 환자군에서폭로 odd = 대조군에서폭로 odd = = 2 = 0.5 Odds ratio = = 4

78

79

80 Selection bias in Case-Control Study

81 커피음용비율 (%) 환자군대조군일반인

82

83

84 Selection bias in Case-Control Study

85 Selection bias in cohort studies Immigrative selection bias Selection into the cohort is affected both by exposure (or by a cause of exposure) and by risk of disease Emigrative selection bias Selection out of the cohort (losses to follow-up) are affected both by exposure (or by a cause of exposure) and by risk of disease 85

86 86

87 Honolulu Heart Study Rate of CHD, stroke and total mortality in 11,136 men of Japanese ancestry eligible in ,006 men examined and 3,130 not examined (60% completed a mailed questionnaire) Baseline smoking as current, ever or never All 11,136 followed through

88 Differential smoking by participation Examined Unexamined N = 8,006 N = 3,130 Smoking: Ever 69% 72% Current 48% 53% Age-standardized %, p value < 0.01 Selection into cohort associated with smoking 88

89 Non-participants Participants Benfante R, et al. Am J Epidemiol 1989;130:

90 Rate ratios for current smoking Examined (Study) N = 8,006 Un-Examined N = 3,130 Source N = 11,136 Total CVD Stroke

91 Grady D et al, JAMA 2002;288:49-57

92 Grady D et al, JAMA 2002;288:49-57

93 CHD events since randomization HERS and HERS II Grady D, et al. JAMA 2002;288:

94 Psaty BM, et al. JAMA 2001;285:

95 Psaty BM, et al. JAMA 2001;285:

96 Prothrombin variant G A, HRT, and non-fatal MI Population based case-control study Psaty BM, et al. JAMA 2001;285:

97 Selection bias in cohort studies of HRT and CVD Postmenopausal women with prothrombotic mutations may die or have CVD early after onset of treatment These women would not be included in cohort studies of HRT and CVD Source population Exposed Disease No disease Study population Disease No disease A B Exposed b a C D Nonexposed Nonexposed c d 97

98 Selection bias in cohort studies of HRT and CVD HRT Selection Outcome Prothrombotic mutations

99 Psaty BM, et al. JAMA 2001;285:

100 Samaha FF, et al. N Engl J Med 2003;348:

101 Samaha FF, et al. N Engl J Med 2003;348:

102 102

103 Selection bias due to losses of follow-up in RCT of Atkins diet Assigned diet Selection Outcome (weight loss) Age, other factors

104 Minimizing selection bias Random sampling from source population Limit losses to follow up Sensitivity analysis 104

105 Healthy Worker Effect

106 Dibbs E, et al. Circulation 1982;65:

107 107

108 From causal effect to data Phillips CV. Epidemiology 2003;14:

109 Measurement error can affect Exposure Outcome Confounders Mediators Modifying factors 109

110 Error components Measured value = True value + Error Error = Bias + Random Error Systematic component of the error Random component of the error

111 Quantification of measurement error Dichotomous variables Sensitivity, specificity Kappa statistic Categorical variables Spearman correlation coefficient Kappa statistic Continuous variables Coefficient of variation Intraclass correlation coefficient (reliability coefficient) 111

112 Differential vs. non-differential errors Non-differential measurement error Measurement error in the variable in question (e.g., the exposure) does not depend on the levels of other variables (e.g., the outcome, confounders, etc) Differential measurement error Measurement error depends on the levels of other variables (for instance, when sensitivity and specificity for measuring disease are different in exposed and unexposed participants) 112

113 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 100% Observed RR = 1.71 Yes Diseased No Exposed Yes No

114 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 100% Specificity = 90% Observed RR = 1.91 Yes Diseased No Exposed Yes No

115 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 90% Observed RR = 1.60 Yes Diseased No Exposed Yes No

116 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure In this case, measurement error will induce a bias will be towards the null, unless The test is uninformative or misleading The true effect is null The magnitude of the bias depends on: Sensitivity and specificity The prevalence of the exposure The risk of the disease The magnitude of the true effect The measure of association used 116

117 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure 117

118 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

119 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

120 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

121 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED DISEASE Sensitivity = 80% Specificity = 90% Observed RR = 1.41 Yes Diseased No Exposed Yes No

122

123

124

125

126

127

128

129 Regression towards the mean When a variable is measured with random error and we select participants with observed extreme values, their true underlying values are on average closer to the population mean Measure BP and select participants with SBP > 140 mmhg Consequences Inconsistencies in diagnosis and classification Biases in evaluation of interventions Inefficiency in planning studies 129

130 Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED Yes DISEASE Sens in exposed = 90% Sens in unexposed = 80% Spec in exposed = 100% Spec in unexposed = 100% Observed RR = 2.25 Diseased No Exposed Yes No

131 Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED Yes DISEASE Sens in exposed = 100% Sens in unexposed = 100% Spec in exposed = 90% Spec in unexposed = 80% Observed RR = 1.00 Diseased No Exposed Yes No

132 Differential measurement error Can bias measures of association in any direction The magnitude can be substantial, even with small differences in sensitivity or specificity In cohort studies, an important concern is differential classification of disease as a function of exposure Diagnostic bias Surveillance bias Mask follow-up procedures and outcome assessment 132

133 Main points on measurement error Measurement errors are pervasive in epidemiological studies Non-differential, independent errors in exposure or outcome tend to bias associations towards the null, but there are exceptions Differential or dependent errors can bias the association in either direction If sensitivity / specificity or ICC are known from validation studies, we can correct the measures of association 133

134 Strategies for increasing accuracy of measurements Standardize measurement methods in an operations manual Train and certify observers Refine the instruments Automate instruments and procedures Calibrate equipment Make unobtrusive measurements Blind measurements Take repeated measurements

135 Generalizability (external validity) 135

136 PPV Information Sheet 136

137 Generalizability is a judgment Consider if the same biological / social mechanisms apply in the target population as in the source population Consider if the prevalence of factors that may modify the effect of the exposure are different in the target and in the source population E.g., genetic determinants Be careful 137

138 Any question?