Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

Size: px

Start display at page:

Download "Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University"

Dulcie Gray
5 years ago
Views:

1 Biases in clinical research Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

2 Learning objectives Describe the threats to causal inferences in clinical studies Understand the role of random variability in clinical studies Describe, understand, and learn how to control the 3 main types of bias: Confounding Information bias / measurement error Selection bias Discuss the concept of generalizability of study results DCR Chapters 4 and 9 2

3 Threats to causal inference Truth in the Universe infer Truth in the Study infer Findings in the study Research Question Random and systematic error Study Plan Random and systematic error Actual Study Target Population Intended Sample Actual subjects Design Implementation Phenomena of interest Intended variables Actual measurement s EXTERNAL VALIDITY INTERNAL VALIDITY 3

4 Bias Systematic difference between the true value and the measured value How close is the measured value to the true value? Synonym for bias: lack of validity Validity on average, the measurement estimates the true measurement

5 Real example of random error! Body weight, bathroom scale True body weight 180 lbs Inconsistent scale, but set correctly at 0 lbs Moments apart: 1 st measurement: lbs 2 nd measurement: lbs 3 rd measurement: lbs 4 th measurement: lbs

6 Real example of bias! Body weight, bathroom scale True body weight 180 lbs Consistent scale, but fail to set it at 0 lbs: reads -5 lbs Moments apart: 1 st measurement: 175 lbs 2 nd measurement: 175 lbs 3 rd measurement: 175 lbs 4 th measurement: 175 lbs

7 Threats to causal inference Lack of precision Random variability - by chance We may observe an association that does not exist or may fail to observe an existing association Lack of internal validity Bias - Systematic errors Confounding Information bias / measurement error Selection bias 7

8 Threats to causal inference Incorrect assessment of the direction of causality: We believe that A B But, in reality A B Lack of external validity (generalizability): True effect in the study population But, does not apply to other populations 8

9 Precision vs. validity Meta-analysis of long-term large randomized controlled trials of statins and coronary heart disease endpoints 9

10 Streptokinase in AMI Meta-analysis Lau J, et al. N Engl J Med 1992;327:

11 Smoking and CVD mortality NHANES II Mortality Study Sample size: 9,205 Length of follow-up: 16 years Prevalence at baseline Current smokers: 32.2% Former smokers: 26.8% Never smokers: 41.0% Hazard ratio for all-cause mortality: Current vs. never smokers: 2.08 (95% CI ) Former vs. never smokers: 1.32 (95% CI ) 11

12 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality in random samples of N = 500 Ex-smokers Curr. smk [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

13 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing current to never smokers N = 500 N = 1, N = 5,000 N = 9,205

14 Smoking and CVD mortality NHANES II Mortality Study Hazard ratios for mortality comparing former to never smokers N = 500 N = 1, N = 5,000 N = 9,205 14

15 Bias definition Deviation of results or inferences from the truth Any trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically different from the truth Last JM, ed. A dictionary of epidemiology, 4th ed. Oxford, Oxford University Press,

16 Bias classification Many different biases have been described Sackett DL. Bias in analytic research. J Chron Dis 1979;32:51-63 Delgado-Rodriguez M, Llorca J. J Epidemiol Community Health 2004;58: general types of biases: Confounding Misclassification / Information bias Selection bias 16

17 From causal effect to data 17

18 Direction of Bias (I)

19 Direction of Bias (II)

20 From causal effect to data

21 Confounders are factors that Cause the disease (or are surrogates for causal factors) AND Have a different distribution in exposed and unexposed populations (i.e., are associated with the exposure in the study sample) Both conditions need to be present to have confounding We will also need the additional condition that the confounder is not affected by the exposure

22 Confounders have to 1. Cause the disease (or be a surrogate measure of a cause) AND 2. Be associated with exposure (i.e., be distributed differently between exposed and unexposed), AND 3. Not affected by exposure (i.e., not be an intermediate variable in the causal pathway) Note: the 3 conditions are necessary for a variable to be a confounder 22

23 Concepts of confounding MI No MI Coffee No coffee Odd ratio (OR)= Smokers Nonsmokers MI No MI MI No MI Coffee No Coffee OR in smokers= OR in nonsmokers=

24 Concepts of confounding

25 Concepts of confounding

26 Concepts of confounding

27 Concepts of confounding

28 Concepts of confounding

29 Confounding 29

30 Asking about sex 30

31 Comparability of exposure groups 31

32 Sex and mortality Results 32

33 Sex and mortality Recommendations! 33

34 Causal associations Low sex frequency Death from myocardial infarction Low sex frequency Death from myocardial infarction Poor health

35 35

36 Hypotheses 36

37 Methods 37

38 Results and interpretation 38

39 Causal diagram Sexual behavior HIV infection AIDS Use of amyl nitrite 39

40 Confounders are factors that Cause the disease (or are surrogates for causal factors) AND Have a different distribution in exposed and unexposed populations (i.e., are associated with the exposure in the study sample) Both conditions need to be present to have confounding We will also need the additional condition that the confounder is not affected by the exposure

41 Amyl nitrite, HIV infection, and AIDS Sexual behavior HIV infection AIDS Use of amyl nitrite HIV infection causes AIDS HIV infection and use of amyl nitrite were associated in homosexual men 41

42 Confounders have to 1. Cause the disease (or be a surrogate measure of a cause) AND 2. Be associated with exposure (i.e., be distributed differently between exposed and unexposed), AND 3. Not affected by exposure (i.e., not be an intermediate variable in the causal pathway) Note: the 3 conditions are necessary for a variable to be a confounder 42

43 Causal diagram Physical activity, HDL cholesterol, and MI Low physical activity Low HDL cholesterol Myocardial infarction Low physical activity is a cause low HDL cholesterol Low HDL cholesterol is a cause of myocardial infarction HOWEVER, low HDL cholesterol is an intermediate variable in the causal pathway between physical activity and myocardial infarction 43

45 Uncontrolled confounding Unmeasured confounders Unknown confounders Known confounders that are too expensive or difficult to measure Residual confounding Confounder is measured imperfectly, and cannot be controlled completely 45

46 Results and interpretation 46

47 Confounding by indication 47

48 In practice (I) Prior knowledge on the biological and other causal relationships is needed to properly identify which variables to adjust for Do NOT apply statistical criteria to decide if the conditions for confounding are present Testing for the association of confounder with exposure and of confounder with disease Stepwise selection procedures Consider if exposed and unexposed subjects are comparable with respect to their risk of disease (except for exposure) 48

49 In practice (II) Consider which determinants of disease may be responsible for the lack of comparability Elaborate causal diagram Identify causal factors that may be different between exposed and unexposed Obtain information on potential confounders Measuring confounders with error will result in residual confounding after adjustment Use statistical techniques to adjust for potential confounders 49

50 Methods to control for confounding In the design of the study Randomization Restriction Matching primarily in case-control studies In the analysis Standardization Stratification Multivariate models Propensity scores Inverse probability weighting Sensitivity analysis 50

51 Enrollment and follow-up in HERS Grady D, et al. JAMA 2002;288:

52 52

53 53

54 54

55 Restriction Restrict eligibility into the study to one category of the confounder Advantages Convenient, inexpensive and easy to implement Adequate control of confounder Simple analysis Disadvantages Cannot evaluate effect modification May limit generalizability For continuous variables, need to group the restriction variable (possibility of residual confounding) Can only be applied to a small number of variables 55

56 Restriction to lifetime non-smokers to avoid confounding by smoking

57 57

58 From causal effect to data 58

59 Selection bias The measure of association observed in the study sample is different to the measure of association in the source population Selection into the study is affected both by the exposure (or by a cause of the exposure) AND by a cause of the outcome (in cohort studies) or by the outcome (in case-control studies) Source population Study population Exposed Disease No disease Disease No disease A B Exposed b a C D Nonexposed Nonexposed c d 59

60 (1000 ) (2000 ) Risk = = 0.01 Risk = = risk 0.01 Relative risk = = risk = 4

61 (1000 ) (2000 ) : 15 ( Hx 10, Hx 5 ) : 15 ( Hx 10, Hx 5 ) odd = odd = = 2 = 2 Odds ratio = 2 2 = 1

62 1) Fat intake? : multi-center :

63 (1000 ) (2000 ) : 15 ( Hx 10, Hx 5 ) : 15 ( Hx 1, Hx 14 ) odd = odd = = 2 = Odds ratio = = 28

65 (1000 ) (2000 ) : 15 ( 10, 5 ) : 15 ( 5, 10 ) odd = odd = = 2 = 0.5 Odds ratio = = 4

67 MacMachon B, et al. N Engl J Med 1981;304:630-3

68 MacMachon B, et al. N Engl J Med 1981;304:630-3

70 Selection bias in Case-Control Study

71 (%)

74 Selection bias in Case-Control Study

75 Selection bias in cohort studies Immigrative selection bias Selection into the cohort is affected both by exposure (or by a cause of exposure) and by risk of disease Emigrative selection bias Selection out of the cohort (losses to follow-up) are affected both by exposure (or by a cause of exposure) and by risk of disease 75

76 Healthy Worker Effect

77 77

78 78

79 79

80 Honolulu Heart Study Rate of CHD, stroke and total mortality in 11,136 men of Japanese ancestry eligible in ,006 men examined and 3,130 not examined (60% completed a mailed questionnaire) Baseline smoking as current, ever or never All 11,136 followed through

81 Differential smoking by participation Examined Unexamined N = 8,006 N = 3,130 Smoking: Ever 69% 72% Current 48% 53% Age-standardized %, p value < 0.01 Selection into cohort associated with smoking 81

82 82

83 Rate ratios for current smoking Examined (Study) N = 8,006 Un-Examined N = 3,130 Source N = 11,136 Total CVD Stroke

84 Samaha FF, et al. N Engl J Med 2003;348:

85 Samaha FF, et al. N Engl J Med 2003;348:

86 86

87 Selection bias due to losses of follow-up in RCT of Atkins diet Assigned diet Selection Outcome (weight loss) Age, other factors

88 Minimizing selection bias Random sampling from source population Limit losses to follow up Sensitivity analysis 88

89 From causal effect to data Phillips CV. Epidemiology 2003;14:

90 Measurement error can affect Exposure Outcome Confounders Mediators Modifying factors 90

91 Error components Measured value = True value + Error Error = Bias + Random Error Systematic component of the error Random component of the error

92 Quantification of measurement error Dichotomous variables Sensitivity, specificity Kappa statistic Categorical variables Spearman correlation coefficient Kappa statistic Continuous variables Coefficient of variation Intraclass correlation coefficient (reliability coefficient) 92

93 Differential vs. non-differential errors Non-differential measurement error Measurement error in the variable in question (e.g., the exposure) does not depend on the levels of other variables (e.g., the outcome, confounders, etc) Differential measurement error Measurement error depends on the levels of other variables (for instance, when sensitivity and specificity for measuring disease are different in exposed and unexposed participants) 93

94 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 100% Observed RR = 1.71 Diseased Yes No Exposed Yes No

95 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 100% Specificity = 90% Observed RR = 1.91 Diseased Yes No Exposed Yes No

96 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED EXPOSURE Sensitivity = 80% Specificity = 90% Observed RR = 1.60 Diseased Yes No Exposed Yes No

97 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure In this case, measurement error will induce a bias will be towards the null, unless The test is uninformative or misleading The true effect is null The magnitude of the bias depends on: Sensitivity and specificity The prevalence of the exposure The risk of the disease The magnitude of the true effect The measure of association used 97

98 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure 98

99 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

100 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

101 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring exposure

102 Non-differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED DISEASE Sensitivity = 80% Specificity = 90% Observed RR = 1.41 Diseased Yes No Exposed Yes No

103

104

105

106

107

108

109

110 Regression towards the mean When a variable is measured with random error and we select participants with observed extreme values, their true underlying values are on average closer to the population mean Measure BP and select participants with SBP > 140 mmhg Consequences Inconsistencies in diagnosis and classification Biases in evaluation of interventions Inefficiency in planning studies 110

111 Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED DISEASE Diseased Yes No Exposed Yes No Sens in exposed = 90% Sens in unexposed = 80% Spec in exposed = 100% Spec in unexposed = 100% Observed RR =

112 Differential measurement error Dichotomous exposure & outcome Errors in measuring disease TRUE TABLE N = 2000 P(E) = 50% P(D E ) = 10% RR = 2.0 Diseased Yes No Exposed Yes No TABLE WITH MISCLASSIFIED DISEASE Diseased Yes No Exposed Yes No Sens in exposed = 100% Sens in unexposed = 100% Spec in exposed = 90% Spec in unexposed = 80% Observed RR =

113 Differential measurement error Can bias measures of association in any direction The magnitude can be substantial, even with small differences in sensitivity or specificity In cohort studies, an important concern is differential classification of disease as a function of exposure Diagnostic bias Surveillance bias Mask follow-up procedures and outcome assessment 113

114 Main points on measurement error Measurement errors are pervasive in epidemiological studies Non-differential, independent errors in exposure or outcome tend to bias associations towards the null, but there are exceptions Differential or dependent errors can bias the association in either direction If sensitivity / specificity or ICC are known from validation studies, we can correct the measures of association 114

115 Strategies for increasing accuracy of measurements Standardize measurement methods in an operations manual Train and certify observers Refine the instruments Automate instruments and procedures Calibrate equipment Make unobtrusive measurements Blind measurements Take repeated measurements

116 Generalizability (external validity) 116

117 PPV Information Sheet 117

118 Generalizability is a judgment Consider if the same biological / social mechanisms apply in the target population as in the source population Consider if the prevalence of factors that may modify the effect of the exposure are different in the target and in the source population E.g., genetic determinants Be careful 118

119 Any question?

Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University

Biases in clinical research. Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University Biases in clinical research Seungho Ryu, MD, PhD Kanguk Samsung Hospital, Sungkyunkwan University Learning objectives Understand goal of measurement and definition of accuracy Describe the threats to causal